400 likes | 576 Views
Acoustics’08. The role of auditory-visual integration in object recognition. Clara Suied 1 , Nicolas Bonneel 2 and Isabelle Viaud-Delmon 1 1 CNRS – UPMC UMR 7593 Hôpital de la Salpêtrière, Paris, France 2 REVES / Inria Sophia-Antipolis, France.
E N D
Acoustics’08 The role of auditory-visual integration in object recognition Clara Suied1, Nicolas Bonneel2 and Isabelle Viaud-Delmon1 1CNRS – UPMC UMR 7593 Hôpital de la Salpêtrière, Paris, France 2REVES / Inria Sophia-Antipolis, France Research supported by the EU IST FP6 Open FET project CROSSMOD
Recognition of natural object • Recognizing a natural object involves pooling information from various sensory modalities • And to ignore information from competing objects How do these multisensory information interact to form a unique object concept?
Object recognition • To direct action to objects, spatial information needs to be encoded and this might interact with object perception • For recognition tasks with the spatial dimension not relevant to the task, conflicting results (Gondan et al., 2005; Teder-Salerjarvi et al., 2005) • Realistic object are of interest in the study of multisensory integration, since a given object can be identified through any of several single modalities • Little behavioural studies with realistic objects (e.g. Molholm et al., 2004 for an ERP study; Laurienti et al., 2004 for linguistic-type stimuli)
Main experiment: Object Recognition • Identification task: go/no-go • When the target (telephone) is either heard or seen, press the button as fast as possible • Withold response when distractor (frog) is presented alone
RING RING • Unimodal • Bimodal semantically congruent • Bimodal semantically incongruent V+ A+0 A+40 RING RING CROAK CROAK RING RING A+0V+ A+0V- A+40V+ A+40V- A-0V+ A-40V+ Go conditions
Unimodal • Bimodal semantically congruent CROAK CROAK CROAK CROAK V- A-0 A-40 A-0V- A-40V- No-Go conditions
Experimental questions • Spatial alignment necessary for fast object recognition? • Larger auditory-visual integration for realistic objects? • Effect of distractors (semantic congruence) on performance?
Bimodal Visual target Auditory target Results
Experimental questions • Spatial alignment necessary for fast object recognition? • Larger auditory-visual integration for realistic objects? • Effect of distractors (semantic congruence) on performance?
Spatial alignment Bimodal Visual target Auditory target
Spatial alignment • 2 (spatial alignment) x 4 (conditions) repeated-measures ANOVA • Main effect of the spatial alignment (F1,19=17.68; p<0.0005) • Main effect of the condition (F3,57=65.36; ε= 0.8; p<0.0001) • But NO INTERACTION • the spatial effect is a Stimulus-Response Compatibility (Simon and Craft, 1970; Simon et al., 1981; Lu and Proctor, 1995) Spatial alignment does not facilitate object recognition
Experimental questions • Spatial alignment necessary for fast object recognition? • Larger auditory-visual integration for realistic objects? • Effect of distractors (semantic congruence) on performance?
Auditory-visual integration p < 0.0001
AV integration and not statistical facilitation Race Model (Miller, 1982)
Size of the AV integration • Computation of the effect size of the AV integration observed in the A+0V+ condition (Cohen’s d; Cohen, 1988) • Comparison with the size of AV integration previously observed in the literature where
Experimental questions • Spatial alignment necessary for fast object recognition? • Larger auditory-visual integration for realistic objects? • Effect of distractors (semantic congruence) on performance?
p < 0.005 Role of a distractor on object recognition Auditory distractor Visual distractor
Role of a distractor on object recognition • When the distractor is visual • No performance cost when processing an auditory target • When the distractor is auditory • There is a performance cost when processing a visual target It seems impossible to ignore an auditory distractor
Conclusion • Large bimodal integration effect • Size of the visual object, realism, 3D and large display, immersive • No effect of spatial alignment on object recognition • Spatial alignment important for saccade generation or signal detection (Stein and Meredith, 1993; Hughes et al., 1994; Frens et al., 1995; Harrington and Peck, 1998) • Object recognition is a function where spatial alignment is not essential It could reflect the fact that this function probably involves brain regions containing neurons with broad spatial receptive fields • A possible asymmetry in the attentional filtering of irrelevant auditory and visual information • Similar asymmetry for cueing effect in detection tasks (Schmitt et al., 2000) • Alerting role of the auditory system?
AV integration and not statistical facilitation Race Model (Miller, 1982)