Recalibration of Audiovisual Simultaneity by Adaptation to a Constant Time Lag

1. VSS Third Annual Meeting May 10, 2003 Recalibration of Audiovisual Simultaneity by Adaptation to a Constant Time Lag Waka Fujisaki1, Shinsuke Shimojo1,2, Makio Kashino1 & Shin’ya Nishida1 1.NTT Communication Science Laboratories, NTT Corporation 2.California Institute of Technology

2. Background • Audiovisual Interactions - tighter than previously thought （e.g.,Driver, 1996, Sekuler et al., 1997, Recanzone, 1998, Shams et al., 2000, Watanabe & Shimojo, 2001, Kitagawa & Ichihara, 2002 ） • Cue to Bind Audiovisual Events → Audiovisual Simultaneity • Perception of Audiovisual Simultaneity • The point of subjective simultaneity • The range (the temporal window) of subjective simultaneity -１-

3. Audiovisual Simultaneity : Problem (Tappe et al., 1994; Stone et al., 2001; Sugita & Suzuki, 2003) Subjective Time  Compensation of Subjective Simultaneity Subjective Time  Neural Transmission & Processing Time Course of Neural Events Input Subjective Simultaneity How can brain get correct answer (physical simultaneity) ? -2-

4. How can brain get correct answer? Derived from same event! Time lags with high cross correlation Simultaneity! Two possibilities Audiovisual subjective simultaneity is • Fixed • Adaptively Recalibrated -3-

5. Feasibility of ‘Adaptive Recalibration’ Theory • Adaptive recalibration in other domains • Within-modal recalibraion of simultaneity (Bennett & Westheimer, 1985; Okada & Kashino, 2003) • Cross-modal recalibration in the spatial domain: ventriloquism aftereffect (Canon, 1970; Recanzone, 1998） • Audiovisual subjective simultaneity may also be adaptively shifted -4-

6. Prediction: adaptation to the constant time lags Subjective Simultaneity (Before Adaptation) Subjective Simultaneity (After Adaptation) Input Transmission & Processing Time Course of Neural Events Subjective Time Shifts of audiovisual subjective simultaneity -5-

7. Method Test of audiovisual simultaneity -412〜+412ms Adaptation to a constant audiovisual time lag -647〜+647ms 5 deg. white ring （<12 ms） 1800 Hz tone pip （about 10 ms） -6-

8. Method • Construction of Each Session Initial Adaptation (3 min) Top-up Adaptation (10 s) Test 78 trials(test points (13)×number of trials (6) ) -7-

9. Method • Tasks • Test • Judge audiovisual simultaneity (3 choices)SimultaneousNot simultaneous (related) Not simultaneous (not related) • Judge temporal order (2 choices) Auditory stimulus first Visual stimulus first • Adaptation • Detect odd stimuli (smaller ring, lower tone pip) appear in 5% of total stimuli 　→To attract subjects’ attention to adapters -8-

10. Estimation of Point of Subjective Simultaneity(Gaussian Fitting) Point of Subjective Simultaneity Simultaneity Responses 1 r > 0.92 0 -500 -250 0 250 500 Time Lags of Test Points(ms) -9-

11. ResultPoint of Subjective Simultaneity (3 subjects) AM SN* WF* 30 100 -25 15 50 -50 0 0 -75 -15 -50 -100 Adapted Time Lags -30 -100 -125 -400 -400 -400 0 0 0 400 400 400 Point of subjective simultaneity shifted in the direction to reduce the adapted lag Light First Point of Subjective Simultaneity (ms) Sound First Light First Time Lags of AdaptedConditions (ms) Error bar: 95% confidence interval * : Authors -１1-

12. ResultPoint of Subjective Simultaneity (7 subjects) 4/7 6/7 P=.07 P<.01 P<.01 7/7 Adapted time lags Light First 300 200 The center point of subjective simultaneity significantly shifts 100 Point of Subjective Simultaneity (ms) 0 -100 -200 Error bar：1SE -300 -235 0 +235 Light First Sound First Adapted Conditions (ms) -１2-

13. Range of Subjective Simultaneity What happens to the range of subjective simultaneity when the point of subjective simultaneity shifts？ -１3-

14. Range of Subjective Simultaneity What happens to the range of subjective simultaneity when the point of subjective simultaneity shifts？ ①distribution shifts？ Before Adaptation ② range extends? Before Adaptation -１3-

15. Range of Subjective Simultaneity What happens to the range of subjective simultaneity when the point of subjective simultaneity shifts？ ①distribution shifts？ After Adaptation ② range extends? After Adaptation -１3-

16. Range of Subjective Simultaneity What happens to the range of subjective simultaneity when the point of subjective simultaneity shifts？ ①distribution shifts？ Before Adaptation ② range extends? Before Adaptation -１3-

17. Range of Subjective Simultaneity What happens to the range of subjective simultaneity when the point of subjective simultaneity shifts？ ①distribution shifts？ After Adaptation ② range extends? After Adaptation -１3-

18. Estimation of the Range of Subjective Simultaneity Range of Subjective Simultaneity (half height width) 1 Lower Boundary Upper Boundary 50% of amplitude 0 -500 -250 0 250 500 Time lags of test stimuli(ms) -１4-

19. ResultRange of Subjective Simultaneity (7 subjects) Time lags of adapters were out of the range of subjective simultaneity Upper boundary Lower boundary Adapted time lags Light First 300 200 100 The range significantly extends towards the adapted lag Range of Subjective Simultaneity (ms) 0 -100 -200 Error bar：1SE -300 -235 0 +235 Sound First Light First Adapted Conditions (ms) -１5-

20. Conclusion • The effects of adaptation to the audiovisual time lags • Point of subjective simultaneity shifted in the direction to reduce adapted lag • Range of subjective simultaneity extended towards the adapted lag Audiovisual subjective simultaneity is adaptively recalibrated -１6-

21. Level of Adaptation • Peripheral? • Unlikely, since the effect remains to occur even when adaptation and test stimuli differ in visual display type and stimulated ear. • Change in Response Bias? • More than that, since implicit / indirect simultaneity decision reflected in the temporal tuning of the stream / bounce illusion is similarly influenced by the adaptation. (cf. Nishida et al.,2003, ECVP)

22. How long adaptation is necessary? • It seems that initial adaptation (3 min) was enough to develop adaptation. Point of Subjective Simultaneity (frame) blocks (6 blocks in one session) Changes in the point of subjective simultaneity during an experimental session

23. Why A→V was as effective as V→A ? Light First • A→V is ecologically unlikely, but the adaptation effect was almost symmetrical. Why? • Because the brain learned that A→V is not unlikely at least in our experimental room. • Symmetrical adaptation effects are consistent with our proposal that the brain uses adaptation to recalibrate simultaneity from the temporal relationship that occurs most frequently. Asymmetrical effects (e.g., suppression of the adaptation to the A→V time lag) would suggest that the brain knows simultaneity beforehand. 400 200 Range of Subjective Simultaneity (ms) 0 -200 -400 -235 0 +235 Sound First Adapted conditions (ms)

24. Perception of Adapters • Were adaptation pairs perceived as simultaneous?

25. ResultRange of Subjective Simultaneity　（7 subjects） Light First Time lags of adapters were out of the range of subjective simultaneity 400 200 Adaptation pairs were NOT perceived as simultaneous, yet effective as adaptation stimuli Range of Subjective Simultaneity (ms) 0 -200 -400 -235 0 +235 Sound First Light First Adapted conditions (ms)

26. ResultRange of ‘Not Simultaneous But Related’ Time lags of adapters were out of the range of subjective simultaneity Adaptation pairs were NOT perceived as simultaneous, yet effective as adaptation stimuli Adaptation pairs were perceived as ‘related’ Light First 400 200 Range of Subjective Simultaneity (ms) 0 -200 -400 -235 0 +235 Sound First Adapted conditions (ms)

27. ResultPoint of Subjective Simultaneity (3 subjects) AM SN* WF* 30 100 -25 15 50 -50 0 0 -75 -15 -50 -100 Adapted Time Lags -30 -100 -125 -400 -400 -400 0 0 0 400 400 400 Point of Subjective Simultaneity (ms) Time Lags of Adapted Conditions (ms)