Synchronization of musical sound and visual images: Issues of empirical and practical significance in multimedia develop

1. Synchronization of musical sound and visual images:Issues of empirical and practical significance in multimedia development Scott D. Lipscomb, Ph.D. Institute for Music Research, University of Texas at San Antonio Acoustical Society of America

2. Acoustical Society of America

3. Acoustical Society of America

4. Acoustical Society of America

5. Acoustical Society of America

6. Acoustical Society of America

7. Model of Film Music PerceptionLipscomb & Kendall (1994) Acoustical Society of America

8. Association Judgment Referential Meaning Acoustical Society of America

9. Influential Musical Parameters: • clarity of tonal center • harmonic complexity • dynamic variation • tempo • absolute rate & fluctuation • phrase structure • melodic activity Acoustical Society of America

10. Accent Structure Alignment Syntactical Meaning Acoustical Society of America

11. Sources of Perceived Accent • Musical • pitch height, loudness, timbre • Visual • spatial location, shape, color Acoustical Society of America

12. Consonant Out-of-Phase Dissonant Alignment Possibilities Acoustical Society of America

13. Method - AV Stimuli • Experiment One • simple single-object animations and pitch sequences (5-sec); created by author • Experiment Two • excerpts from experimental animations by Norman McLaren (8-sec); “Dots” (1940), “Canon” (1964), & “Synchromy” (1971) • Experiment Three • excerpts from “Obsession” (25-sec) with musical score by Bernard Herrmann Acoustical Society of America

14. Acoustical Society of America

15. Acoustical Society of America

16. Control & Reliability vs.Ecological Validity Experiments Two & Three Acoustical Society of America

17. Experiment Two “Dots” by Norman McLaren Acoustical Society of America

18. Experiment Two “Synchromy” by Norman McLaren Acoustical Society of America

19. Experiment Two - VAMECollapsed Across Alignment Condition Acoustical Society of America

20. Experiment Three “Obsession”--music by B. Herrmann Acoustical Society of America

21. Experiment Three - SynchronizationCollapsed Across Alignment Condition Acoustical Society of America

22. Experiment ThreeEffectiveness - Combined Groups Acoustical Society of America

23. Comparison of VAME Results Across All Three Experiments Acoustical Society of America

24. VAME - SynchronizationAcross All Experiments Acoustical Society of America

25. VAME - EffectivenessAcross All Experiments Acoustical Society of America

26. Perception Accent Structure Alignment Association Judgment No Shift of Attentional Focus Shift of Attentional Focus Film Music Perception Paradigm(revised) Aural Stimulus Visual Stimulus Audio-Visual Congruence Implicit Processes No Yes Acoustical Society of America

27. Enhancing Instructional Materials with Multimedia CD-ROM Companion to the Handbook of Music Psychology (D. Hodges, Ed.; IMR Press) Acoustical Society of America

28. Hyperlinks to Sound Files “If a pianist plays a low A (110 Hz) on the piano, the resulting vibration does not consist of only a single, periodic vibration at the rate of 110 times per second. Rather, the string vibrates as a whole (110 Hz), in halves (220 Hz), in thirds (330 Hz), in fourths (440 Hz), and so on. Therefore, there is vibrational energy not only at the fundamental frequency of 110 Hz but also at each of these integer multiples [audio examples: a) complex tone made up of 8 partials & b) complex tone built one partial at a time …” Lipscomb & Hodges, 1996, p. 97 Acoustical Society of America

29. Sounds to Attract Attention “If a pianist plays a low A (110 Hz) on the piano, the resulting vibration does not consist of only a single, periodic vibration at the rate of 110 times per second. Rather, the string vibrates as a whole (110 Hz), in halves (220 Hz), in thirds (330 Hz), in fourths (440 Hz), and so on. Therefore, there is vibrational energy not only at the fundamental frequency of 110 Hz but also at each of these integer multiples [audio examples: a) complex tone made up of 8 partials & b) complex tone built one partial at a time …” Lipscomb & Hodges, 1996, p. 97 Acoustical Society of America

30. Exploratory Environments • Beating—experimentation with constructive & destructive interference and the critical bandwidth • WaveMix—combining partials to create a complex tone • Signal—allows student to create a complex signal by manipulating partial amplitudes & phase relationships, hear the result, and view an animation approximating molecular motion Acoustical Society of America

31. Sound as a Cue • Computer Guts • instructional package designed by Dr. David Sebald (UTSA Division of Music) • more examples available online at: http://www.aim-ed.com/ Acoustical Society of America

32. Conclusions: Use of Multimedia to Enhance the Learning Experience • provide additional information through illustration • audio examples or animations • focus user attention • explicit association judgments • create exploratory environments • allowing students the opportunity to incorporate the highest levels of knowledge (Bloom, 1956) a work in progress Acoustical Society of America

33. Contact Info:Dr. Scott D. LipscombInstitute for Music Research Division of Music The University of Texas at San Antonio 6900 N. Loop 1604 West San Antonio, TX 78249 phone: (210) 458-5334 FAX: (210) 458-4381 lipscomb@utsa.edu http://music.utsa.edu/~lipscomb Acoustical Society of America