Communicating in a Natural Cocktail Party: Relating Human and Animal Behavior to Neural Response

1. Communicating ina Natural Cocktail Party:Relating Human and AnimalBehavior to Neural Response Barbara Shinn-Cunningham Boston University Auditory Neuroscience Laboratory

2. Michele Dent Liz McClaine Kamal Sen Rajiv Narayan Erol Ozmeral Erick Gallun Gin Best With funding from ONR, AFOSR, & NIH

3. The everyday acoustic environment is full of competition and clutter • The “Cocktail Party Problem” (Cocktail Party by SLAW, Maniscalco Gallery)

4. Penguins and other social birds suffer from the cocktail party problem Penguins recognize their mates and offspring amidst thousands of birds Chicks identify parents from 11 m -- when call is 6 dB below the level of the background noise (thanks to M. Dent)

5. Zebra finches learn to make a call by listening to a tutor… while in a large colony • Zebra finches are a model system for studying • vocal production learning • hierarchical encoding of complex signals (e.g., “birds own song” neurons; Narayan et al., 2006)

6. How do we figure out what is in the world from the sound mixtures we hear? • time (sec) • air pressure

7. How do we figure out what is in the world from the sound mixtures we hear? • time (sec) • Syllables / words heard as units • Confusions occur between sources (streaming over time) • air pressure

8. Frequency analysis breaks sound into parallel channels • time (sec) • mechanical vibration in air • neural firing (electrical spikes) in auditory nerve • low • When sounds overlap in their spectral content, neural responses are a mixture • air pressure • high

9. Spectrotemporal structure of sound is critical(in contrast with simple, “traditional” stimuli) • Supports segregation of competing source “units” • harmonicity • common onsets • comodulation • Reduces likelihood of spectrotemporal overlap (important elements are unlikely to be masked) • Moreover, theimportant information is contained in the spectrotemporal structure of sound. • BUT… • removing linguistic/semantic effects may tease out different contributing mechanisms

10. This project uses birdsongs from male zebra finch • Can compare results with avian behavior (Dent lab) and neurophysiological responses (Sen lab) • Spectrotemporal structure supports segregation

11. Listeners were trained to identify individual bird songs Moe Uno Toro Nibbles Junior

12. A quick test for you… • What is this? • What is this? Moe Uno Toro Nibbles Junior

13. Short-term “units” segregate, but streaming errors occur for similar sources • What is this? • + Moe Nibbles • What is this? Moe Uno • + Uno noise Toro Nibbles Junior

14. Three maskers, to tease apart different types of interference

15. Three maskers, to tease apart different types of interference

16. Three maskers, to tease apart different types of interference

17. Main type of interference Effect of spatial separation Reduce audibility Improve audibility through acoustic better-ear effects ? ? Cause streaming confusions Allow spatial attention to combat confusions Why these maskers? Spectrotemporalstructure DenseMuch overlap Noise More sparseTemporal control for chorus Mod. Noise SparseKey target features audible Chorus

18. separated • co-located Better-ear effects: the Target-to-Masker Energy Ratio improves with separation • separated • co-located

19. separated • co-located Better-ear effects: the Target-to-Masker Energy Ratio improves with separation • separated • co-located

20. Binaural effects: Interaural decorrelation causes masked signal to be audible • Running cross-correlation output for 500-Hz channel • (simple model of brainstem processing in Medial Superior Olive)

21. Binaural effects: Interaural decorrelation causes masked signal to be audible • Running cross-correlation output for 500-Hz channel • (simple model of brainstem processing in Medial Superior Olive) • Important for signals below about 1500 Hz, but the birdsongs have a lot of high-frequency information

22. Hypothesized role of spatial attention in complex settings ?

23. Measure performance with and without spatial separation of target / masker Masker Masker

24. M M Quantify spatial unmasking = improvement in threshold Co-located Separated

25. M Spatial Unmasking M M - M Quantify spatial unmasking = improvement in threshold Co-located Separated

26. M M For birdsong, better-ear energy effects are large No better ear Better ear

27. M Is there an additional benefit of perceived separation? Diotic versus binaural Diotic

28. M Is there an additional benefit of perceived separation? Diotic versus binaural Diotic • Better-ear benefit • No binaural processing - Sources perceived at same location

29. Binaural • Better-ear benefit • Maybe binaural processing - Sources perceived at different locations M M Is there an additional benefit of perceived separation? Diotic versus binaural Diotic • Better-ear benefit • No binaural processing - Sources perceived at same location

30. Main type of interference Effect of spatial separation Diotic vs. binaural performance? Reduce audibility Improve audibility through acoustic better-ear effects Identical ? ? ? Cause streaming confusions Allow spatial attention to combat confusions Binaural much better than diotic performance Why these maskers? Noise Mod. Noise Chorus

31. For co-located target/masker, the chorus causedthe most interference worse performance (need louder target) Target threshold (dB re: masker target)

32. Spatial separation causes unmasking due to better-ear acoustics (diotic presentation) improvement with spatial separation Identify Target Best et al. 2005

33. Size of acoustic effect decreases as masker becomes sparser (audibility less of a problem) Spatial separation causes unmasking due to better-ear acoustics (diotic presentation) improvement with spatial separation Identify Target Best et al. 2005

34. No advantage from spatial attention or binaural processing (high-frequency content) For dissimilar maskers,there is no added benefit of perceived spatial separation improvement with spatial separation Identify Target Best et al. 2005

35. Perceived separation adds 10 dB of spatial unmasking, for confusable masker No advantage from spatial attention or binaural processing (high-frequency content) For a chorus masker, perceived location differences improve identification improvement with spatial separation Identify Target Best et al. 2005

36. … … … … … • LEDs on the speakers: • no information • which speaker • which time • or both Ask listener to identify a song from a random location, occurring at a random time Five simultaneous, similar sources, every 15 deg

37. For best subjects, when cue less important; they report “pop out” of familiar songs For identification of familiar birdsongs in a chorus,when and where bothhelp

38. For identifying digits intime-reversed digits,when doesn’t help For all subjects, when cue less important; forward digits “pop out” of reversed speech

39. Ongoing work How prior knowledge affects spatial attention The role of visual cuing of spatial attention Divided auditory attention Comparisons with visual attention Modeling spatial release from different interference

40. Psychophysics shows different maskers cause different forms of perceptual interference • Noise and modulated noise • reduce audibility of song elements • are dissimilar from targets • are easily segregated • don’t cause confusion • show spatial release due to acoustic better-ear improvements • Chorus (or reversed speech) • is sparse enough that overall interference is not as great • consists of “units” (syllables) like those in the targets • is hard to segregate from target • causes confusion between target and masker • shows spatial release through spatial attention

41. Comparing to avian behavior Dent Lab SUNY Buffalo

42. Dent Lab: Teaching birds to recognize the songs PECK left key to begin variable waiting period (2-7 s) HEAR a call from one of six individuals RECOGNIZE and CATEGORIZE call PECK left or right key Correct: food reward Incorrect: lights extinguished

43. Zebra finch and budgerigars learn zebra finch songs Percent Correct *Average sessions to criterion = 34.25 = 22

44. Relative effectiveness of the maskers differs across species worse performance (need louder target) Target threshold (dB re: masker target)

45. Next stage: measuring whether the birds use spatial attention like humans Budgerigars exhibit spatial release for noise maskers Binaural Hearing S + N 35 Unilateral Sound Source Bilateral Sound Source 30 25 20 Masked Threshold (in dB) 15 10 S + N N 5 0 1.00 2.00 2.86 4.00 Frequency (in kHz) *Dent et al., Behav. Neurosci., 1997

46. Avian psychophysics shows different maskers cause different levels of interference • Degree of interference differs from species to species • Next stage will explore whether effect of spatial separation differs with masker type, as in humans

47. Comparing to avian physiology Sen Lab Biomedical Engineering

48. Recording from zebra finch forebrain Field L (homologue of primary auditory cortex) Record neural spike trains in response to multiple copies of clean songs from five birds Record neural spike trains in response to repetitions of each song embedded in each masker

49. Each neuron has a set of spectrotemporal features to which it responds Broadband onset neuron Narrowband neuron Frequency Time

50. Compare clean-song templates to target + masker