Pitch , tonality, and the missing fundamentals of music cognition

1. Pitch, tonality, and the missing fundamentals of music cognition Richard Parncutt University of Graz, Austria BRAMS, Université de Montréal 31 May 2012 This file has been revised after discussion and questions SysMus Graz

2. Abstract What are the psychological foundations of major-minor tonality? Psychologists have explored how modern listeners perceive its pitch structures, but the psychohistorical origins of those structures remain unclear. A plausible theory should be able to predict tonal styles as probability distributions of pitch-time patterns on the basis of a limited number of psychologically and historically plausible axioms. From a psychological viewpoint, such axioms should refer to pitches that are perceived (experienced) by audiences and performer - not pitches notated in scores. Non-notated pitches may include prominent partials, missing fundamentals, or pitches expected on the basis of short- or long-term experience (e.g. melodic continuations). Consider a simple example that ignores octave register and tuning. A C-major triad may have a missing fundamental at A, because E corresponds to the 3rd harmonic of A and G to the 7th. Other possible missing fundamentals are F and D. The same chord may have a prominent partial at B, if the 3rd harmonic of E and the 5th harmonic of G coincide; another prominent partial may be at D. A systematic approach should consider all such possibilities in a chord’s spectrum, weighting them relative to each other. Predicted pitches and weights should be consistent with empirical data. But the psychological reality of non-notated pitches remains unclear because “nature” (predictions based on psychoacoustics or physiology) and “nurture” (predictions based on musical experience) are often quantitatively similar. I will present recent data and plans for future work to separate nature from nurture by systematically manipulating musical expertise, cultural background, sound type, tone type, onset synchrony, duration, tuning and background noise. Further strategies include separation of “fundamental listeners” (sensitive to missing fundamentals) from “spectral listeners” (sensitive to prominent partials), and modeling musical experience by statistical analysis of symbolic music databases.

3. Origin of major-minor tonal system Scientific approach: Psychologically predetermined? • Underlying principles? • Why those pc-sets? voicings? progressions? • Can we model frequency of occurrence? Humanities approach: Historical accident? If so: • Why so widespread? • Why so stable?

4. Assumption The major-minor system is based on pitch as subjective experience • not as physical measurement (frequency) • not as physiological correlate • not as notation in musical scores . Thesis To understand the major-minor system, we must systematically investigate pitch as experienced by musicians and listeners in musical contexts.

5. Does experience exist? Visual experience is quite different from • physical world • info on the retina (upside down, moving) • neurophysiology of the visual cortex Visual experience is constructed • availableinfoisgenerallyincomplete • focus on affordances (survival and reproduction) Correlates of the color red ≠ red itself • light wavelengths • physiology of retina • physiology of visual cortex To study “red”, we must separate experience & physics

6. Is everything physical? Modern science is atheist - ok • Good arguments against existence of gods and spirits Conscious experience is something else! • Different from gods/spirits AND brain substrates • Emerges in infancy, disappears when we die • Foundation of arts and aesthetics The solution: Epiphenomenalism • Experience is a byproduct of neural substrates • Both experience and its substrates exist • Twosidesofthe same coin, paradoxicallyinseparable • Consistent with both neuroscience and philosophy

7. What is “more real”? Objective answer: The physical world It exists without experience - but not vice-versa Existence of experience depends on physical world Subjective answer: Experience Without it we would know nothing (not be human) Existence of physical world depends on experience (“Objective”: subject ≠ object, “Subjective”: s=o) Conclusion No idea. Can’t compare totally different things

8. Why scientists reject experienceand why some humanities scholars reject it too Scientific belief system • Success of modern physics • In inherent superiority of objectivity • Reductionism (belief in simple explanations) • Grouping of mind-body dualism with theism Humanities-science conflict • “Othering” humanities to construct own identity • Refusal to accept own subjectivity (fear?) • Competitive neoliberal researchstructures • Scientists too arrogant, insecure or busy for philosophy

9. ThreemusicalrepresentationsandaspectsofmusicalpitchstructureyoucanexplainwiththemThreemusicalrepresentationsandaspectsofmusicalpitchstructureyoucanexplainwiththem • Physical: Frequenciesandamplitudes • Roomandinstrumentacoustics, roughness 2. Experiential: Pitches andtheirsalience • Timbre, fusion, chordroots, harmonicfunction, harmonictonality 3. Abstract: Notes in musicalscores • Performance, composition

10. The “three worlds” of Karl PopperThe broader context of music representations(not “worlds”) 1. Physical • environment, body, brain 2. Experiential • sensations, emotions 3. Abstract • knowledge, info, culture • Assumption: • A clear separation of 3 representations can clarify discussions of nature and origin of • musical structure • human consciousness

11. Literature on ecological and evolutionary psychology versus consciousness & subjective experience Gallagher, S. & Zahavi, D. (2010). Phenomenological Approaches to Self-Consciousness. Stanford Encyclopedia of Philosophy (online) Gulick, R. van (2004). Consciousness. Stanford Encyclopedia of Philosophy (online) Miller, G. (2007). Reconciling Evolutionary Psychology and Ecological Psychology: How to Perceive Fitness Affordances. ActaPsychologicaSinica 39, 546-555.

12. What I mean by “pitch” • Subjectiveexperience – likethecolorred • One-dimensional • Property of pure/complextones, noise (+tinnitus) • May beambiguousand multiple • Depends on listener, temporal context Here: pitch = perceivedpitch In musictheory: pitch = notatedpitch What I mean by “chroma” • Octave-generalisedperceived pitch • not D4 or D5 - just D • Like pitch class, but experienced – not notated

13. Tone types • Pure tone sinusoidal function of air pressure against time • Complex tone simultaneity of pure tones in any frequency relationship • Harmonic complex tone (HCT) Complex tone whose frequencies correspond to a harmonic series

14. The harmonicseries • equally spaced on a linear frequency scale (e.g in Hz) • unequally on a log frequency scale (e.g. in semitones) • Compared to 12-tone equaltemperament: • 7th harmonicis 1/3 semitone flatter than a m7 above 4th • 11th harmonicismidwaybetween P4 and TT above 8th

15. Spectral versus virtualpitchPitch perceptionaccordingtoTerhardt youtube church bells Spectralpitch (SP) • pitchof a pure tone • pitchof an audible partial of a complex tone • hum tone of a church bell (1s after hammer) Virtual pitch (VP) • pitchof a complex tone • mostconsciouslynoticedpitches in everydaylife • strike tone of a church bell (hammerhitting bell) • pitchatmissing fundamental (e.g. voice on telephone)

16. Spectral versus virtualpitch This distinctionis • ecological based on interactionwiththeenvironment • not physiological based on peripheralandcentralprocessing The ultimateaimispsychophysical: understandtherelationshipbetween soundandexperience

17. Whataboutneurophysiology? We don’t know the functional relationship between neural states and processes and conscious experience • Unique nature of this problem! Never solved (ordid I miss thenews?) • Enormous no. of neurons and connections! Whichstates/eventscorrespondtoexperience?

18. Spectralvs temporal processing Along auditory pathways,we find both • temporal representations (phase locking) • spectral representations (tonotopicstructures) Assumptions • Both are used by neural networks • Both are inextricable in hidden layers Conclusion • Doesn’t help us understand pitch as experience

19. Neuralprocessingofpitchin musicandspeech Bharucha, 1987 • The same neural net canprocess… • spectral and temporal patterns • pitch in speech and music

20. Virtual objects in vision and hearing Gestalt principle of closure – filling the gaps in a familiar pattern Virtual object (Kanizsa, 1955) Incomplete triangle Completed by virtual contours missing fundamental (f0) Auditory image (Bregman, 1990; McAdams, 1984) Incomplete harmonic series Completed by virtual pitch overtones SPL frequency

21. Pitch perception: Experimentalmethod • Listeneradjustsfrequency of pure toneuntil the twosounds have the same pitch • Frequency of pure toneis a measure of pitch of test sound • Results must beconsistent within and betweenlisteners Pitch salience = probability of matching

22. Pitch ambiguity Assumption: The pitch of a pure tone • is unambiguous • corresponds to frequency (if SPL constant) Result: The pitch of a complex tone • is ambiguous = different pitch in different presentations • and/or multiple = several pitches perceived simultaneously  Can explain a lot about musical structure

23. Pitch salience high pitch salience • In musical practice: • Pitched versus unpitched percussion • How clear is pitch on a continuous scale? • In experimental data: • Probability of noticing a pitch • Subjective clarity of a pitch • Depends on: • Stimulus (esp. spectrum) • Listener (“spectral” vs “fundamental”) • Temporal context (proximity  expectation) low pitch salience

24. Analytic versus holistic perception • You can consciously switch between two modes • analytic (strange black shapes) • holistic (“FLY” in white letters) • Similarly for pitch?

25. Individual differences in pitch perception Auditory ambiguity test (Seither-Preisler) Individual differences  “fundamental listeners” and “spectral listeners”

26. AuditoryAmbiguity Test (AAT)Seither-Preisler et al. (2007) You will hear 10 tone pairs In each pair, doesthepitchriseor fall? Write youranswersasarrows: ↑pitchrises ↓pitch falls

27. If you wrote this, you are a “fundamental listener“ If the opposite, you are an “overtone listener” You may also be a “mixed listener”

28. Finding: Listening strategy depends on music experience and instrument Research idea: Study relation to amusia? Schneider et al., NY Acad Sci, Vol. 1060, p. 387-395 (2005) fundamental listeners overtone listeners

29. Pitchdominanceregions Virtual pitch According to model predictions, VP salience is highest at D4 (C2-C6). f0 range of voice and music f1 f2 middle C • Spectral pitch According to experimental data, • SP salience is highest at F5 (C4-C8). •  speech intelligibility & formants: • f1 ~ 500 Hz ~ C5, f2 ~ 1500 Hz ~ G6

30. Dominance region of spectral pitchorigin: speech perceptioncentre at 700 Hz, central band at 300-2000 Hz after Terhardt et al., 1982

31. Calculated VP salience distribution f0 range of speech and music After Huron & Parncutt (unpublished)

32. Origin of virtual pitcha bit of history Before the 1970s many assumed... • low pitch = combination tone = distortion product • peripheral origin (basilar membrane) In the 1970s it became clear... • pitch perception = pattern recognition • mixture of spectral and temporal processing • central origin (brain)

33. PerceptionofcomplextonesTwoseparablestages 1. Auditory spectral analysis  c. 16 audible* or 8 resolvable* harmonics 2. Holistic perception  (Virtual) pitch, timbre, loudness *Audible: If you change it, the listener hears something *Resolvable: Listener can focus attention on it 2 1

34. Did you hear a bee buzzing in your ear?trials and tribulations of recorder ensemble performance ? • Combination tones become audible: • high frequencies, high amplitudes • little low-frequency masking • Origin: Non-linear distortion in inner ear

35. Perceptual fusion of HCTsdepends on: • Tuning of partials Mistuning of <1 semitone from harmonic series • Relative amplitude of partials Is spectral envelope like a typical environmental sound? • Temporal context Preceding/following tones can attract attention • Listener • Fusion more likely for “holistic” or “fundamental” listeners

36. Pitch at the missingfundamentalASA AuditoryDemonstrations CD (Houtsma, Rossing, Wagenaars), track 37 1 • Conclusions: • 1. Pitch does not necessarily correspond to a partial • 2. Pitch is multiple/ambiguous • VP atmissingfundamental • SP atlowest partial 2 3 4 5

37. Sound demo: Masking SP and VPASA AuditoryDemonstrations CD (Houtsma, Rossing, Wagenaars) • Conclusion: • Masking and pitch pattern recognition happen in different places • Maskingisperipheral • Pitch pattern recognition is central 1st tone in pair 2ndtone in pair track 40 41 42

38. Relation between VP and SP pattern ASA AuditoryDemonstrations CD (Houtsma, Rossing, Wagenaars), Track 39 VP correspondsto: • best-fit subharmonicof all partials • NOT frequencydifference • smallmistuningisnoproblem

39. General relation between SP and VP • VP lies at fundamental of audible harmonic pattern • VP salience depends on SPs at harmonic positions • how many there are (the more the better) • their salience (the greater the better) • their tuning (mistuning up to a semitone) • their effective harmonic numbers (the lower the better)

40. Prevalence of individual tones (scale steps) in chant Source: LiberUsualis • 1,900 pages; most versions of ordinary chants for the catholic mass • first edited in 1896 by Solesmes abbot Dom André Mocquereau • Online search by CIRMMT: DDMAL (Ichiro Fujinaga and team) no. of notes counted A B C D E F G

41. Prevalence of individual tones (scale steps) in chant Source: LiberUsualis • 1,900 pages; most versions of ordinary chants for the catholic mass • first edited in 1896 by Solesmes abbot Dom André Mocquereau • Online search by CIRMMT: DDMAL (Ichiro Fujinaga and team) • Accidentals are ignored, but less than 1% of Bs are B=-flats no. of notes counted A B C D E F G

42. Prevalence of individual tones (scale steps) in chant How can we explain the distribution? • Musical structure depends on non-notated chroma This is just one example • Listeners have a “feel” for pitches of harmonics Or at least spectral listeners do • Tones are preferred if consonant with context An example of pitches in common (“pitch commonality”) • Up to ten harmonics are audible (resolvable?) Almost no masking from other sounds

43. Prevalence of individual tones (scale steps) in chant 1. “Octave-generalise” the harmonic series 2. How many “octave-generalised overtones” correspond to diatonic scale?

44. Prevalence of individual tones (scale steps) in chant Data Model A B C D E F G A B C D E F G df = 5, r = 0.90, p<.01 cf. Parncutt, R. & Prem, D. (2008). The relative prevalence of Medieval modes and the origin of the leading tone (poster). International Conference on Music Perception and Cognition (ICMPC10), Sapporo, Japan, 25-29 August.

45. Guillaume de Machaut(1300-1377) Rondeau Ma fin est mon commencement What is the origin of (rising) leading tones? Why do rising semitones “tonicize”?

46. This is not a popular theory! Music psychologists: • No “cognitive structures” • Empirical evidence is unclear (BUT: consistent with statistical learning) Psychoacousticians and neuroscientists: • Focuses on subjective experience • Avoids temporal-spectral debate Music theorists: • Challenges primacy of musical score • Focuses on tonality (not “modernist”) Music historians: • Not based on historic sources • Ignores historic mode classification • Contradicts… • physicalmonism • establishedresearchparadigms in sciences and humanities

47. Non-notated chroma in triadsAn example of looking carefully at the stimulus (for a change) 1. Spectral synthesis Build a C major triad from • first 10 harmonics of C4 (up to E7) • harmonics of E4 and G4 (up to F#7) Assume chromatic categorical perception 2. Masking Assume all partials are equally audible except inside a chromatic cluster 3. Pitch pattern recognition At each chromatic scale step: • Which harmonics are present in chord? • Synthesize that tone using “SFS Esynth”

50. Estimating virtual pitch salience Compromise between • simplicity (parsimony, falsifiability) • accuracy (accounting for all factors) First approximation • Count the audible harmonicsaboveany pitch (nextslide) Second approximation • Weighteachharmonic 1/n, thenaddweights (slideafterthat) Closer approximations • Estimateaudibility of partial, normalise salience (Parncutt, 1989) • Considertuning of partials (Terhardt et al., 1982) • Consider spectral dominance region (Terhardt et al., 1982)