1 / 39

Perceiving Transformational Invariance Discovering the basis multi-sensory integration

Explore the mechanisms behind speech perception and multi-sensory integration. Discover how the brain integrates input signals and the role of temporal synchrony, cortical convergence zones, and co-modulation. Also, learn about the McGurk effect and the integration of audio, visual, and tactile information.

phillipsj
Download Presentation

Perceiving Transformational Invariance Discovering the basis multi-sensory integration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Perceiving Transformational InvarianceDiscovering the basis multi-sensory integration Richard Hurtig Department of Speech pathology & Audiology The University of Iowa

  2. Speech Perception is the result of Sensory Integration Sign wave speech

  3. Speech Perception is hard! Massaro, 1998

  4. It is not all in the acoustic signal:Stop-Gap Trading Relation [s_lit split] Repp 1985

  5. Top Down EffectsWarren Phoneme Restoration “BABY” [ba_y] + [__clap_] [ba:clap:y] “baby” with clap in the background

  6. What tells the brain that what the eyes are seeing goes with what the ears are hearing and what the skin is feeling? Possible Solutions: Temporal Synchrony of Input Signals Mapping to Cortical Convergence Zones Co-Modulation of Primary Sensory Areas of Cortex

  7. Scrots Perspective makes things look ‘normal’

  8. Fresco Perspective allows Forms to emerge

  9. Circular ProjectionsSkull in the Mirror

  10. AnamorphismWhen do thingslook right?

  11. Recognizing Faces/ObjectsDepends on Perspectiveand Prior Experience

  12. Monica

  13. Look the same? Thompson, P. (1980). “Margaret Thatcher: a new illusion”, Perception, 9, 483-484

  14. Spectral Inversion

  15. Upside Down Environmental Sounds Saw Bird Kiss Doorbell Rooster Cow Toilet Dog

  16. Upside Down Speech & Music

  17. Image Recognition of Distorted Images:Digital Image Morphing Monroe Madonna Elvis

  18. How Much of the signal do we need? Low Pass High Pass

  19. Filtered Music & Speech Low Pass <2000 Band Pass 1500-3000 High Pass >2500 “The doctor prescribed the drug”

  20. “Play it Backwards”How important is temporal order? Dog Saw Bird Cow Toilet Kiss Rooster Doorbell

  21. Reversed Speech & Music Edgar Allen Poe

  22. Under What Conditions Do We RecognizeTemporal Inversion? • Palindromes/ Symmetry • Segmental Elements • BOB • STATS • Metrical Schemes • 000- - - 000

  23. Size/Gender

  24. Frequency ComprehensionTime Domain and Frequency DomainRepresentations of the syllable /ma/Blue=OriginalRed=Compressed(75%)

  25. Spectral CompressionTFT ALGORITHM(Hurtig & Turner, Patented 2003) Array Remapping FFT IFFT Input Signal Output Signal A/D D/A 50 % Compression 4 3 3 2 2 1 1 50% compression Original 4-tone

  26. Compressed Environmental Sounds 80% Dog Saw Rooster Cow Bird Doorbell Kiss Toilet

  27. Compressed Speech & Music 80% 100%

  28. Compressed Singing Voice:Perception of Gender and Size Soprano Soprano Tenor Bass Original 90% 80% 70%

  29. Multi-Modal Speech PerceptionMcGurk Effect DA GA BA

  30. Multi-Modal Speech PerceptionMcGurk Effect- 2 BDA/BGA BA GA

  31. Synthesized & NaturalMcGurk Effect BA VA THA DA Fixed Audio Variable Video Fixed Video Variable Audio Massaro, 1998

  32. Perceive [my dad taught me to drive]See [my gag gog me goo grive]Hear [my bab bob me boo brive] Massaro, 1998

  33. Gesture-Speech ExperimentBa-Da Continuum Massaro, 1998

  34. Speech Perception In Noise Multi-modal effects Audio Only SNR +5 dB Audio & Visual SNR +1 dB Audio & Tactile SNR +3 dB

  35. Where does the integration take place? (Posner & Raichle , 1994)

  36. What does increased rCBF say about functional modules? (Posner & Raichle , 1994)

  37. Why There Can’t Be A Pre-wired Module Uniquely Reserved for Language In The Brain • How would we account for multi-modal effects? • Why would the module have evolved a somato-sensory input? • Sur & Leamey (2001) Rerouting visual inputs to auditory cortex

  38. Current Projects • Utilizing multi-sensory integration to assess Hearing Aid benefit- with Ruth Bentler • Implementing a spectral compression algorithm in a wearable sensory aid- with Chris Turner

  39. Contact Information • Richard Hurtig • Department of Speech Pathology & Audiology, College of Liberal Arts & Sciences • The University of Iowa • Iowa City, Iowa 52242 • 319-335-8730 • Richard-Hurtig@uiowa.edu

More Related