1 / 39

The role of auditory-visual integration in object recognition

Acoustics’08. The role of auditory-visual integration in object recognition. Clara Suied 1 , Nicolas Bonneel 2 and Isabelle Viaud-Delmon 1 1 CNRS – UPMC UMR 7593 Hôpital de la Salpêtrière, Paris, France 2 REVES / Inria Sophia-Antipolis, France.

astra-herman
Download Presentation

The role of auditory-visual integration in object recognition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Acoustics’08 The role of auditory-visual integration in object recognition Clara Suied1, Nicolas Bonneel2 and Isabelle Viaud-Delmon1 1CNRS – UPMC UMR 7593 Hôpital de la Salpêtrière, Paris, France 2REVES / Inria Sophia-Antipolis, France Research supported by the EU IST FP6 Open FET project CROSSMOD

  2. Recognition of natural object • Recognizing a natural object involves pooling information from various sensory modalities • And to ignore information from competing objects How do these multisensory information interact to form a unique object concept?

  3. Object recognition • To direct action to objects, spatial information needs to be encoded and this might interact with object perception • For recognition tasks with the spatial dimension not relevant to the task, conflicting results (Gondan et al., 2005; Teder-Salerjarvi et al., 2005) • Realistic object are of interest in the study of multisensory integration, since a given object can be identified through any of several single modalities • Little behavioural studies with realistic objects (e.g. Molholm et al., 2004 for an ERP study; Laurienti et al., 2004 for linguistic-type stimuli)

  4. Main experiment: Object Recognition • Identification task: go/no-go • When the target (telephone) is either heard or seen, press the button as fast as possible • Withold response when distractor (frog) is presented alone

  5. RING RING • Unimodal • Bimodal semantically congruent • Bimodal semantically incongruent V+ A+0 A+40 RING RING CROAK CROAK RING RING A+0V+ A+0V- A+40V+ A+40V- A-0V+ A-40V+ Go conditions

  6. Unimodal • Bimodal semantically congruent CROAK CROAK CROAK CROAK V- A-0 A-40 A-0V- A-40V- No-Go conditions

  7. Experimental questions • Spatial alignment necessary for fast object recognition? • Larger auditory-visual integration for realistic objects? • Effect of distractors (semantic congruence) on performance?

  8. Bimodal Visual target Auditory target Results

  9. Experimental questions • Spatial alignment necessary for fast object recognition? • Larger auditory-visual integration for realistic objects? • Effect of distractors (semantic congruence) on performance?

  10. Spatial alignment Bimodal Visual target Auditory target

  11. Spatial alignment • 2 (spatial alignment) x 4 (conditions) repeated-measures ANOVA • Main effect of the spatial alignment (F1,19=17.68; p<0.0005) • Main effect of the condition (F3,57=65.36; ε= 0.8; p<0.0001) • But NO INTERACTION • the spatial effect is a Stimulus-Response Compatibility (Simon and Craft, 1970; Simon et al., 1981; Lu and Proctor, 1995) Spatial alignment does not facilitate object recognition

  12. Experimental questions • Spatial alignment necessary for fast object recognition? • Larger auditory-visual integration for realistic objects? • Effect of distractors (semantic congruence) on performance?

  13. Auditory-visual integration

  14. Auditory-visual integration p < 0.0001

  15. AV integration and not statistical facilitation Race Model (Miller, 1982)

  16. Size of the AV integration • Computation of the effect size of the AV integration observed in the A+0V+ condition (Cohen’s d; Cohen, 1988) • Comparison with the size of AV integration previously observed in the literature where

  17. Large AV integration

  18. Experimental questions • Spatial alignment necessary for fast object recognition? • Larger auditory-visual integration for realistic objects? • Effect of distractors (semantic congruence) on performance?

  19. p < 0.005 Role of a distractor on object recognition Auditory distractor Visual distractor

  20. Role of a distractor on object recognition • When the distractor is visual • No performance cost when processing an auditory target • When the distractor is auditory • There is a performance cost when processing a visual target It seems impossible to ignore an auditory distractor

  21. Conclusion • Large bimodal integration effect • Size of the visual object, realism, 3D and large display, immersive • No effect of spatial alignment on object recognition • Spatial alignment important for saccade generation or signal detection (Stein and Meredith, 1993; Hughes et al., 1994; Frens et al., 1995; Harrington and Peck, 1998) • Object recognition is a function where spatial alignment is not essential It could reflect the fact that this function probably involves brain regions containing neurons with broad spatial receptive fields • A possible asymmetry in the attentional filtering of irrelevant auditory and visual information • Similar asymmetry for cueing effect in detection tasks (Schmitt et al., 2000) • Alerting role of the auditory system?

  22. AV integration and not statistical facilitation Race Model (Miller, 1982)

More Related