Informed Source Separation of Orchestra and Soloist Using Masking and Unmasking

1. ISCA Tutorial and Research Workshop on Statistical And Perceptual Audition (SAPA) 2010 Informed Source Separation of Orchestra and Soloist Using Masking and Unmasking By Yushen Han, Christopher Raphael School of Informatics and Computing, Indiana University Bloomington Saturday 25 September 2010, Makuhari, Japan

2. Motivation – Musical Source Separation In General • To extract the orchestra accompaniment from any desirable recordings given the score - can be used in automatic accompaniment system (e.g. a piano concerto) or Karaoke • To isolate one chosen instrument from an ensemble • can be used in performance analysis of soloist

3. Motivation – This Project • Problem: separation procedures cause damage to each of the separated sources • Object: to address the degradation of the separation results • Strategy: by exploiting the information redundancy of the musical audio within each source

4. Overview • Motivation and introduction – system diagram • Previous – Separation by Spectrogram Masking • Recent – Repair by Spectrogram Unmasking • harmonicity hypothesis tested by Kalman phase smoothing • repair by amplitude inference and harmonic transposition • Examples with evaluation by spectrogram reassignment • Relevant works – ISS? • Conclusion and Discussion

5. Informed Source Separation System Diagram System Input System Output • Expectation- Maximization • Dynamic Programming • Machine Learning • (binary classification) • Harmonic- Percussive Separation by Spectrogram Masking Audio-score Alignment Desoled Audio Damaged Audio EM DP ML HPSS Score Evaluation According to BASS Note-wise audio reconstruction Note sample models 2D Spectral Modeling Note sample library Phase Estimation by Kalman Smoothing Harmonicity Hypothesis Amplitude Inference Desoled Audio Repaired Phase Estimation Mostly Previous Work Recent Development Focus of This Paper

6. Score Following

7. Separation “Informed” by Score Following solo accompaniment

8. Previous: (Binary) Spectrogram Masking Short-time Fourier Transform Complementary binary masks with (hard binary mask)

9. Previous: 2D Note-based Model a “template” function qm of note model indexed by m

10. Informed Source Separation System Diagram System Input System Output • Expectation- Maximization • Dynamic Programming • Machine Learning • (binary classification) • Harmonic- Percussive Separation by Spectrogram Masking Audio-score Alignment Desoled Audio Damaged Audio EM DP ML HPSS Score Evaluation According to BASS Note-wise audio reconstruction Note sample models 2D Spectral Modeling Note sample library Phase Estimation by Kalman Smoothing Harmonicity Hypothesis Amplitude Inference Desoled Audio Repaired Phase Estimation Mostly Previous Work Recent Development Focus of This Paper

11. Phase Estimation • Amplitude-Phase Decoupling Model Slowing varying at hth harmonic Locally linear in s up to a small correction term amplitude signal phase Phase unwrapping

12. State-space Model for Phase (cont.) • This idea of using state-space model to estimate phase should be credited to A. TaylanCemgil. For observable phase sequence at hth harmonic we introduce state vector with an unobservable component As timesprogresses (discretely), the state vector propagates via the state equation Where state transition matrix governs the the sinusoidal movement of phase according to the average phase advance at harmonic h over a relative long period (s0, s1) and w(s)is an unobservable, zero-man random (state) perturbation.

13. Illustration of State-space Model for Phase Estimation s s + 1 x2 x1 = (x1(s), x2(s) )t, but only observe y = x1 connects the observed and unobservable where H(s)=[1 0] and r(s) = 0 is the degenerated random (observation) perturbation

14. Kalman Smoothing Follows the state-space model, we can obtain the amplitude and phase This state-space model can be computed by Kalman filter but since the phase estimation is offline, we can update the state estimates backward to incorporate the observation that were not “available” at sample t in the forward pass by Kalman smoothing

15. Informed Source Separation System Diagram System Input System Output • Expectation- Maximization • Dynamic Programming • Machine Learning • (binary classification) • Harmonic- Percussive Separation by Spectrogram Masking Audio-score Alignment Desoled Audio Damaged Audio EM DP ML HPSS Score Evaluation According to BASS Note-wise audio reconstruction Note sample models 2D Spectral Modeling Note sample library Phase Estimation by Kalman Smoothing Harmonicity Hypothesis Amplitude Inference Desoled Audio Repaired Phase Estimation Mostly Previous Work Recent Development Focus of This Paper

16. Phase Estimation And Pairwise Unwrapped Phase Difference ? pitch G#3 (written A#3 on Bb clarinet) over a crescendo clarinet

17. Application of Phase Estimation – Using Pairwise Unwrapped Phase Difference to Test the Harmonicity Hypothesis By projecting the unwrapped phase θi(s) from harmonic i to j we visualize the unwrapped phase difference between harmonics in woodwinds and strings to test the harmonicity hypothesis

18. Informed Source Separation System Diagram System Input System Output • Expectation- Maximization • Dynamic Programming • Machine Learning • (binary classification) • Harmonic- Percussive Separation by Spectrogram Masking Audio-score Alignment Desoled Audio Damaged Audio EM DP ML HPSS Score Evaluation According to BASS Note-wise audio reconstruction Note sample models 2D Spectral Modeling Note sample library Phase Estimation by Kalman Smoothing Harmonicity Hypothesis Amplitude Inference Desoled Audio Repaired Phase Estimation Mostly Previous Work Recent Development Focus of This Paper

19. Amplitude Inference: Sampling a Note

20. Amplitude Inference by Conditional Expectation

21. Informed Source Separation System Diagram System Input System Output • Expectation- Maximization • Dynamic Programming • Machine Learning • (binary classification) • Harmonic- Percussive Separation by Spectrogram Masking Audio-score Alignment Desoled Audio Damaged Audio EM DP ML HPSS Score Evaluation According to BASS Note-wise audio reconstruction Note sample models 2D Spectral Modeling Note sample library Phase Estimation by Kalman Smoothing Harmonicity Hypothesis Amplitude Inference Desoled Audio Repaired Phase Estimation Mostly Previous Work Recent Development Focus of This Paper

22. Damage Repair b b

23. Experiment an excerpt of 45 seconds from the 2nd movement of Ravel’s piano concerto in G major

24. Excerpt from the 2nd movement of Ravel’s piano concerto in G major

25. Relevant works • BSS • NMF (non-negative “part-based representation” in NMF) • Latent variable decomposition by Raj, Smaragdis • Other Score-guided separation by Dubnov • “Informed Source Separation” using watermark by Parvaix • Harmonic/Percussive Sound Separation (HPSS), by Sagayama, Ono • Physical Acoustics, Fletcher

26. Conclusion and Future Work • Harmonic-wise Information Redundancy Expressed In both amplitude and phase can be used to inference “partially” damaged notes • Creating a framework to perform separation/repair in a large scale with synthesized ground truth and with BASS performance measurement by E. Vincent • (coming soon) xavier.informatics.indiana.edu/~yushan/SAPA2010

27. FINE Thank you for your attention