Recent Research in Musical Timbre Perception

1. Recent Research in Musical Timbre Perception James W. Beauchamp University of Illinois at Urbana-Champaign Andrew B. Horner Hong University of Science and Technology Michael D. Hall James Madison University, Harrisonburg, VA

2. Starting Point • Timbre experiments are based on musical instrument sounds.

3. Starting Point • Timbre experiments are based on musical instrument sounds. • Perform short-time spectral analysis.

4. Starting Point • Timbre experiments are based on musical instrument sounds. • Perform short-time spectral analysis. • Identify parameters of ST spectrum:

5. Starting Point • Timbre experiments are based on musical instrument sounds. • Perform short-time spectral analysis. • Identify parameters of ST spectrum: • Partial (harmonic) amplitudes - Time variation - Spectral envelope (centroid, irregularity, etc.)

6. Starting Point • Timbre experiments are based on musical instrument sounds. • Perform short-time spectral analysis. • Identify parameters of ST spectrum: • Partial (harmonic) amplitudes - Time variation - Spectral envelope (centroid, irregularity, etc.) • Partial (harmonic) frequencies - Time variation - Inharmonicity

7. Methods for Studying Timbre Stimuli Preparation In Freq. Domain • Simplification • Perturbation • Normalization

8. Methods for Studying Timbre • Listener Experiments • Discrimination (pairs) • Timbral Distance Estimation • Classification • Identification Stimuli Preparation In Freq. Domain • Simplification • Perturbation • Normalization

9. Methods for Studying Timbre • Listener Experiments • Discrimination (pairs) • Timbral Distance Estimation • Classification • Identification Stimuli Preparation In Freq. Domain • Simplification • Perturbation • Normalization • Data Processing/Presentation • Discrimination (sensitivity) scores/plots • Multidimensional Scaling • Correspondence (R2) Measurements

10. Studies Reviewed • 1999 Discrimination Study • 2006 Discrimination Study • 2006 Multidimensional Scaling (MDS) Study • 2009 Discrimination/Classification Study

11. 1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds • clarinet, flute, oboe, trumpet, violin, harpsichord, marimba

12. 1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds • clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize F0, loudness, and duration.

13. 1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds • clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize F0, loudness, and duration. • Test sounds: Apply six spectrotemporal simplifications.

14. 1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA) • Seven reference sounds • clarinet, flute, oboe, trumpet, violin, harpsichord, marimba • Equalize F0, loudness, and duration.. • Test sounds: Apply six spectrotemporal simplifications. • Subjects discriminate between original and simplified sounds.

15. 1999 Discrimination StudyResults Discrim Score • Spectral envelope smoothing 96% • Spectral flux elimination 91% • Amplitude envelopes smoothing 66% • Frequency envelopes smoothing 70% • Freq. envs. harmonic locking 69% • Frequency variations elimination 71%

16. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Eight sustained musical instrument tones • bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin

17. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Eight sustained musical instrument tones • bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • Modified by fixed random transfer function

18. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Eight sustained musical instrument tones • bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • Modified by fixed random transfer function • F0, loudness, duration, centroid preserved

19. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Eight sustained musical instrument tones • bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin • Modified by fixed random transfer function • F0, loudness, duration, centroid preserved Typical spectral envelopes:

20. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to discrimination between original and modified tones.

21. 2006 Discrimination StudyHorner, Beauchamp, and So JAES • Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to the discrimination data. • Best results: obtained by relative-amplitude (harmonic) spectral error:

22. 2006 Discrimination StudyHorner, Beauchamp, and So JAES Discrimination vs. error level (): R2=0.81

23. 2006 Discrimination Study Horner, Beauchamp, and So JAES Discrimination vs. rel-amp spec error: R2=0.90 for a=1.0

24. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin

25. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin • F0, loudness, duration, attack & decay times, and average spectral centroid are equalized.

26. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin • F0, loudness, duration, attack & decay times, and average spectral centroid are equalized. • Two types of tones: static (flux removed) and dynamic (flux retained).

27. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin • F0, loudness, duration, attack & decay times, and average spectral centroid are equalized. • Two types of tones: static (flux removed) and dynamic (flux retained). • Subjects estimate timbral dissimilarity between instruments.

28. 2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) • Ten sustained musical instrument tones • bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin • F0, loudness, duration, attack & decay times, and average spectral centroid are equalized. • Two types of tones: static (flux removed) and dynamic (flux retained). • Subjects estimate timbral dissimilarity between instruments. • Data processed by two multi-dimensional scaling (MDS) programs (SPSS & Matlab).

29. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: • Even/Odd: Ratio of even and odd harmonic rms amplitudes.

30. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: • Even/Odd: Ratio of even and odd harmonic rms amplitudes • Spectral IRregularity: Degree of jaggedness of a spectrum.

31. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: • Even/Odd: Ratio of even and odd harmonic rms amplitudes • Spectral IRregularity: Degree of jaggedness of a spectrum. • Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value. For Dynamic Tones Only:

32. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: • Even/Odd: Ratio of even and odd harmonic rms amplitudes • Spectral IRregularity: Degree of jaggedness of a spectrum. • Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value. • Spectral INcoherence: Degree of spectral change relative to the average spectrum (same as flux). For Dynamic Tones Only:

33. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Correlations: E/O: R=0.78 SIR: R=0.69 Static Tone Case SPSS algorithm Stress=0.12

34. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Correlations: E/O: R=0.79 SIR: R=0.75 Static Tone Case Matlab algorithm Stress=0.12

35. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Correlations: E/O: R=0.71 SCV: R=0.68 SIN: R=0.56 SIR: R=0.39 Dynamic Tone Case SPSS algorithm Stress=0.17

36. 2006 MDS StudyBeauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Correlations: E/O: R=0.69 SCV: R=0.68 SIN: R=0.53 SIR: R=0.40 Dynamic Tone Case Matlab algorithm Stress=0.15

37. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Goals/Purpose • Exp. 1. Relative importance of spectral vs. temporal cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.

38. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Goals/Purpose • Exp. 1. Relative importance of spectral vs. temporal cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope. • Exp. 2. Relative importance of spectral envelope (formant) structure vs. spectral centroid: Compare discrimination/classification performance for interpolated tones vs. tones obtained by filtration which matches the centroids of the interpolated tones.

39. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 1 Method • Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)

40. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 1 Method • Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) • Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres. Temporal Vn I01 I02 I03 I10 I11 I12 I13 I20 I21 I22 I23 I30 I31 I32 Tr Spectral

41. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 1 Method • Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) • Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres. • Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.).

42. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 1 Method • Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) • Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres. • Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.). • Subjects’ tasks: • 1) to discriminate tone pairs. • 2) to classify tones as ‘violin’, ‘trombone’, or ‘other’.

43. 2009 StudyHall and Beauchamp (Canadian Acoustics) Experiment 1 Results Discrimination: reference stimuli: Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.

44. 2009 StudyHall and Beauchamp (Canadian Acoustics) Experiment 1 Results Classification: Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.

45. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 2 Method • Reference stimulus: Original impoverished violin.

46. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 2 Method • Reference stimulus: Original impoverished violin. • Test stimuli: • 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).

47. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 2 Method • Reference stimulus: Original impoverished violin. • Test stimuli: • 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept). • 2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1).

48. 2009 StudyHall and Beauchamp (Canadian Acoustics) • Experiment 2 Method • Reference stimulus: Original impoverished violin. • Test stimuli: • 1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept). • 2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1). • Subjects’ tasks: Discrimination and classification as in Exp. 1. (Which has the greater effect? Interpolation or filtration?)

49. 2009 StudyHall and Beauchamp (Canadian Acoustics) Experiment 2 Results Discrimination: Classification:

50. Conclusion Summary • 1999 discrimination study: • Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.