Automatic transcription of polyphonic piano music using a note masking technique

1. Automatic transcription of polyphonic piano music using a note masking technique Mr Ronan Kelly and Dr Jacqueline Walker Department of Electronic & Computer Engineering University of Limerick Ronan.kelly@ericsson.com, jacqueline.walker@ul.ie

2. Overview • Music transcription • Our approach • Onset detection • Algorithm • Results • Conclusions

3. Music Transcription • Complex cognitive task Example: Top of the Pops! • A challenging task for a computer but one which pushes boundaries of signal processing, pattern recognition, machine learning,….

4. Monophonic Music Transcription • A solved problem • Sliding window-based analysis of melody line • Steps – decimate – reduce data • Onset detecton • FFT or constant Q transform • Note detection

5. Polyphonic Music Transcription • Multiple simultaneous notes • In Western Tonal Music (WTM), notes played together almost inevitably share harmonics • Impact of rhythms, held notes • Possibility of multiple instruments

6. Approaches to Polyphonic Transcription • Human audition based • Martin Cooke’s “Modelling Auditory Processing and Organisation”, 1993 • Brown & Cooke, “Computational Auditory Scene Analysis”, 1994 • Signal processing based • Tanguiane “Artificial Perception and Music Recognition”, 1993 • Klapuri et al, since 1998

7. Our Approach • Onset Detection • Note Window & FFT • Masking Scheme Iteration

8. 1 ò t = < < ( t ) p ( t ) ( t t t ) , NAE dt + n n 1 t t t n n Onset Detection • NAE (Note Average Energy) Onset detection1. In practice, we search for local minima… 1. (Liu, R., Griffith J., Walker, J. & Murphy, P., TIME DOMAIN NOTE AVERAGE ENERGY BASED MUSIC ONSET DETECTION, Proceedings of the Stockholm Music Acoustics Conference, August 6-9, 2003 (SMAC 03), Stockholm, Sweden

9. Note Window • FFT performed on the whole note • Avoids start-of-note and end-of-note effects • Gives greater robustness against noise

10. Algorithm for Masking Scheme - 1 FFT on note window Find max peak in window Continue until no peaks above threshold Remove peak from window; add to list

11. Algorithm for Masking Scheme - 2 Apply mask to first (lowest) frequency in list Continue until list is empty Adjust amplitudes of all affected frequencies by mask Add frequency to note list; move to next frequency

12. Masking Scheme - 1 C4, E4, G4 262 Hz, 330 Hz, 392 Hz Max. peak amplitude = 29.9 @ 392 Hz (G4) Next peak amplitude = 21.4 @ 330 Hz

13. Masking Scheme - 2 Detected frequency peaks Note mask

14. Masking Scheme - 3 Masking action Note played: C4 After masking

15. Building a Note Mask - 1 A note is played with other notes and the significant frequency peaks and amplitudes recorded: D4 harmonics in common in blue harmonics of D4 in red

16. Building a Note Mask - 2 D4 and A4 D4 and C4

17. Building a Note Mask - 3 Extract values unique to D4 and normalise to amplitude of highest peak:

18. Building a Note Mask - 3 Average across samples:

19. Experimental Set-up • Keyboard used: Technics KN800 PCM Keyboard • Note range: C2 to B6 • Recording – direct using line-in • Isolated chords and polyphonic music samples

20. Results How to define error? Need to account for both missed notes (m) and spurious notes (x) n is number of notes detected – not number of notes played

21. Results – Isolated Chords

22. Results – Polyphonic Music

23. Effect of Onset Detection • Effective onset detection is crucial • Two types of errors: Extra onset less likely to cause a problem but, … note divided up too finely Missing onset note windows not placed ‘correctly’

24. Results with Onset Detection

25. Future Work • Develop model for note combinations (polyphonic note masks) • Use wider range of note combinations • Develop an efficient approach to applying polyphonic note masks • Improve note onset detection