"Theoretical and Experimental Approaches to Auditory and Visual Attention"

1. Ervin Hafter and Anne-Marie Bonnel Department of Psychology U.C.Berkeley “A role for memory in Shared Attention” "Theoretical and Experimental Approaches to Auditory and Visual Attention" Cold Spring Harbor April 20, 2008

2. Simplest definition of attention: a process inferred when responses on one task are affected by responding simultaneously to another

3. Two paradigms S(+) S(+) Detection Identification S(0) S( ) Energy detection model: compare stimuli in the signal epochs S(0) is the level of the standard.

4. Ideal detection based on energy in the signal epochs. 1.00 A S(0) S(+) .90 B Probability density .80 A Area under the ROC λ B S(0) S(+) % correct in 2AFC .70 .60 λ z .50 0.5 1.0 1.5 2.0 2.5 0 Predicts d’B > d’A Signal Levels in z

5. But, Macmillan (w/tonal pedestal) and Bonnel et al. (w/visual pededestal) found d’detection > d’identification

6. Audio demo comparing detection to identification. Stock Market National Debt University salaries

7. Both attributed this to use ofTransients

8. Thinking that, unlike the case with energy, responses to transients might be pre-attentive, Bonnel et al. (1992) tested in a dual task with independent stimuli on side-by-side LEDs.

9. Thinking that, unlike the case with energy, responses to transients might be pre-attentive, Bonnel et al. (1992) tested in a dual task with independent stimuli on side-by-side LEDs. Detection was comparable to performance found with S instructed to attend to only one LED; identification showed a tradeoff indicative of a shared attentional resource.

10. Thinking that, unlike the case with energy, responses to transients might be pre-attentive, Bonnel et al. (1992) tested in a dual task with independent stimuli on side-by-side LEDs. Detection was comparable to performance found with S instructed to attend to only one LED; identification showed a tradeoff indicative of a shared attentional resource. Caveat: “Perceptual grouping” in identification

11. Some changes in Berkeley Visual Auditory S(-) S(0) S(+) CRT 500 Hz 17 ms

12. S(+) Detection Identification S(+) S(0) S( ) S( )

13. Ideal energy detection 1.00 A S(0) S(+) .90 B .80 A Area under the ROC λ B S(0) S(+) % correct in 2AFC Probability density .70 C C .60 λ S(0) .50 S(-) S(+) C 0.5 1.0 1.5 2.0 2.5 C 0 Signal Levels in z 0 -1 -2 +1 +2 λ λ λ z

14. Use the strictest possible criterion for asserting that there is no cost of shared attention: compare performance in the dual-task to that found in the separate single tasks.

15. Support for transient hypotheses: 1) detection >identification 2) No cost in detection 3) Tradeoff in accord with instructions inidentification Attention-Operating Characteristic 50%,50% 3 80%,20% 20%,80% 2 d’audition 50%,50% 80%,20% 1 20%,80% and are single tasks 3 1 2 0 d’vision

16. Sharing-Index Attentional Operating Characteristic (SIAOC) normalizes data from a dual task in terms of the single-task controls. 1.0 .8 d’dual SI = d’single .6 Sharing Index (Audition) .4 .2 1.0 0 .2 .4 .6 .8 Sharing Index Vision) SIAOC

17. These data clearly imply that detection of transience put no demand on shared attention, unlike discrimination of energy. SIAOC 1.0 .8 .6 Sharing Index (Audition) .4 .2 1.0 .2 .4 .6 .8 Sharing Index (Vision)

18. A more direct test of the idea that detecting transience (change per se)doesn’t require shared attention simply removes transients as information.

19. A more direct test of the idea that detecting transience (change per se)doesn’t require shared attention simply removes transients as information.

20. A classic ΔI/I Another classic ΔI/I Often called a reminder

21. Auditory single task no gap gap Energy detection 4.0 Same signal levels 3.0 d’ 2.0 1.0 no 400 600 800 1000 1400 1600 1800 200 1200 gap Gap duration (ms)

22. Dashes suggest absolute identification, i.e. comparisons to long-term, context-coded memory, rather than to a sensory trace of the reminder. 4.0 Same signal levels 3.0 d’ 2.0 1.0 400 600 800 1000 1400 1600 1800 200 1200 Gap duration (ms)

23. represent 500-msec signals. SI-AOC “different” “same” 1.0 “different” .8 .6 “larger” “smaller” .4 .2 < 300 > .2 .6 1.0 .4 .8 Sharing Index (Vision)

24. Impact of increased duration. e.g., , on apparent cost of attention. Test Duration{D} Integration Time {IT} A or V {D} < {IT} Single task D1 = {D} d’1

25. Test Duration{D} Integration Time {IT} A or V {D} < {IT} {D} < {IT} Single task Dual task D1 = {D} D2 = ½ {D} d’2=.707d’1 d’1 d’2=.707d’1 A V

26. Test Duration{D} Integration Time {IT} A or V {D} > {IT} .d’1 Single task D1 = {IT}

27. Test Duration{D} Integration Time {IT} {D} > {IT} .d’1 {D} > {IT} .707d’1< d’2 ≤ d’1 Dual task Single task ½{D} < {D2} < {IT} .707d’1< d’2 ≤ d’1 D1 = {IT} A or V

28. Test Duration{D} Integration Time {IT} A or V {D} > {2IT} .d’1 Single task D1 = {IT}

29. Test Duration{D} Integration Time {IT} A or V {D} > {2IT} .d’1 {D} > {IT} d’2 Single task d’2 D1 = {IT} Dual task {D2} = {IT} *Here, time sharing produces the same result as no sharing.

30. A still stronger test forces the subject to use context-coded memory by simply removing the reminder. “other” “standard” “other” “large” “small”

31. It seems clear that responses based on context-coded memory were limited by sharing of an attentional resource. 1.0 .8 .6 Sharing Index (Audition) .4 .2 .2 .6 1.0 .4 .8 Sharing Index (Vision)

32. Perhaps the reason that use of change per se did not provoke a cost of sharing is that it was done in sensory trace (?rehearsal?)memory?

33. Subjects can be forced to use sensory-trace memory by roving the standard from trial to trial. Correct Trial Rove response Standard Test 1 ‘louder’ 2 ‘softer’ 3 ‘softer’ 4 ‘louder’

34. Unlike the case withcontext-coding, performance fell as the ephemeralsensory-trace faded over time. 3.0 2.5 2.0 da 1.5 Fixed levels Roved levels 1.0 *Need higher signals in roving 0.5 8 3 5 6 4 7 1 2 Auditory Identification GAP Duration (sec) 50-ms reminders and signals

35. 1.0 rove .8 .6 Sharing Index (Audition) <Gap (ms)> Gap (ms) .4 310 .2 510 1022 8350 .2 .6 1.0 .4 .8 Sharing Index (Vision) Most intriguing is that despite very poor performance, especially with long delays, there was no cost of sharing.

36. In typical, everyday life, we label sensory stimuli on the basis of comparisons to long-term memory. Sensory/ Neural Test loud, dim, green, hot, salty, etc. Compare Compare Context-coded, long-term memory Experience

37. When presented with an adjacent standard, the response may be to Sensory/ Neural Sensory/ Neural Test Reminder loud, dim, green, hot, salty, etc. T1 SOA T2 Compare Compare Context-coded, long-term memory Experience

38. When presented with an adjacent standard, the response may be to simply ignore it, labeling the test in accord with long term memory. Sensory/ Neural Sensory/ Neural Test Reminder loud, dim, green, hot, salty, etc. T1 SOA T2 Compare Compare Context-coded, long-term memory Experience

39. Without a reliable context-coded memory, S must compare the test to the reminder the in sensory trace memory. louder, dimmer, greener, hotter, saltier, etc.. Compare Rehearsal Memory Sensory/ Neural Sensory/ Neural Test Reminder loud, dim, green, hot, salty, etc. T1 SOA T2 Compare Compare Context-coded, long-term memory Experience

40. Our audio/video dual-task shows these comparisons to be independent, i.e., no cost of sharing. louder, dimmer, greener, hotter, saltier, etc.. Rehearsal Memory Compare Sensory/ Neural Sensory/ Neural Test Reminder SOA “loud, dim, green, hot, salty, etc.” T1 T2 Compare Compare Context-coded, long-term memory Experience Conversely, these comparisons were limited by a shared attentional resource.

41. Okay, so comparisons to the sensory trace memory produced no cost of shared attention. What in the world is trace memory?

42. Okay, so comparisons to the sensory trace memory produced no cost of shared attention. What in the world is trace memory? Recently, we’ve approached this in terms of Weber’s Law. Bringing the lab up to 1834.

43. Weber’s Law In signal detection terms, k can be treated as a multiplicative noise I = k I

44. Identification Ped = Sig = 50 ms Gap = 1s Without Roving:performance based on based on long-term, labeled memory produce a constant Weber fraction 3 1 -1 Threshold 10 log (ΔI/I) -3 -5 -7 55 60 65 70 75 Pedestal (dB)

45. What happens when comparisons are to a roved standard? Thresholds go up. But in what way? To answer this, we parse the data in terms of the individual standards, analyzing performance separately for each pedestal level.

46. No Rove 3 Labeled memory Rove 1 Trace memory -1 Threshold 10 log (ΔI/I ) -3 -5 -7 55 60 65 70 75 Pedestal

47. I = k I + c The change in slope implies a second, additive noise, c. 3 1 -1 Threshold (dB) -3 multiplicative + additive noise -5 multiplicative noise -7 55 60 65 70 75 Pedestal (dB)

48. 6 4 2 0 -2 -4 -6 55 60 65 70 75

49. ΔI = kI + c What is c? Perhaps it is a decision noise associated with responding to the stimulus in trace memory? Maybe it is simply the result of decay in the trace that makes that adds noise to the remembered amplitude code.

50. Our next plan is to go into fMRI in search of sensory rehearsal. Wish us luck.