Hallucinations in Auditory Perception!!!

1. Hallucinations in Auditory Perception!!! Malcolm Slaney Yahoo! Research Stanford CCRMA

2. Hadoop

5. One Dimensional (waveform) Pressure Time Cochlear Processing Two Dimensional (not a spectrogram) Cochlear Place Time Correlogram Processing Three Dimensional (neural movie) Cochlear Place Time Autocorrelation Lag

6. Correlogram Distance down cochlea Center Frequency Time Interval (s) Autocorrelation Lag With help from Richard O. Duda

7. Correlogram

8. Success Reconstructing from correlogram NIPS Keynote

9. Continuation Tone and Noise Parliament Cough Hear two voices? What do you hear? Waveforms? Ideas? Problems

10. Pressure Time Cochlear Processing Cochlear Place Time Correlogram Processing Cochlear Place Time Autocorrelation Lag

11. Speech Examples Wedding Sine Natural

12. What Vowel is This? Word 1 Word 2 Peter Ladefoged Word 3

16. McGurk

17. Speech Vision Speech Audio Locate Environment Audio Locate Vision Object Speech Speech Object Vision Wedding Vowel? Ventroloquism Dots McGurk Sinewave

20. ASR Three Three Three Language model for the words: “one”, “two”, “three” Two Two Two One One One Word model showing phonemes for the word one /w/ / / /n/ Acoustic (phoneme) model for the phoneme / / S1 S2 S3

21. Conventional Scene Analysis Slide by Dan Ellis (Columbia)

22. Barker—ASR

23. Goto—CASA with MIDI MIDI Sequence

24. Old plus New Principle Slide by Dan Ellis (Columbia)

25. Ellis—Prediction Driven

26. Saliency

27. Saliency Example • Time-frequency display • Saliency map shows high-interest locations

28. Saliency Maps • Longer tones better • Missing parts salient • Modulation more salient • Forward masking works

29. Sound Examples • Birds • Calls • Cows • Horse • Waterfall

30. Saliency Comparison • Details of saliency comparison • Model predictions

31. X Z M Z M M X Y Y m M Relational Network (Simple) • Patches of neurons • Each measureone quantity • Bidirectionalrelations for feedback/feedforward Thanks to Rodney Douglas

32. Relational specification Input here Relational feedback RelationalFeedback Relational Network (example)

33. ASR Relational Network Bidirectional links enforce phoneme/word constraints Phone Recognizer Cochlea Word Recognizer Phone Recognizer Delay A patch of neurons (one of N output) Note: We don’t know how to represent delays

34. Without A A A I Desired Results Relational Feedback With /A/ Phoneme Patch /I/ Phoneme Patch AI Word Patch IA Word Patch Phoneme Input

35. Simulation

36. Simulation 2

37. Simulation 3

38. Grossberg—ART

39. ICA Different distributions One Microphone GMM models of distribution Statistical Means

40. Conventional

41. Better?

42. Thanks malcolm@ieee.org

44. Pitch

45. Silicon Frequency Response • Tone ramps into two cochleas

46. Cochlear Best Frequency

47. Cochlear Rate Profiles Spikes per utterance Left Cochlea Right Cochlea

48. Hardware Overview Phoneme Word Cochlea Learning PCI-AER (for remapping) Learning Cochlea Learning Giacomo Indiveri Shih-Chii Liu PCI-AER (for remapping) Implemented in MATLAB

50. LSH Movie