Audio

1. Audio Sound Audio synthesis IAT 410

2. Audio Perception • Sound: Pressure waves in frequencies between 50Herz - 22,000Herz • Lower frequencies more felt by the whole body than heard • Sounds can be perceived as coming from a location • Not terribly accurate • Cone of confusion IAT 410

3. 3D Audio Perception • Cone of confusion • Cone-shaped zones in front of and behind head • 3D Audio cues: • Interaural Time Difference • Interaural Intensity Difference • Pinnae filtering • Body filtering IAT 410

4. 3D Audio Perception • Goal for 3D sound is “Spatialization” • The sense that the • Sound originates outside your head • Sound has a direction • Interaural Time Difference • The more extremely left or right, the greater the difference • Time difference < 5ms IAT 410

5. 3D Audio Perception • Interaural Intensity Difference • Head absorbs and reflects sound energy • The first ear to get sound gets loudest sound • “Head Shadow” • Cone of confusion: • Time difference too small to detect • Intensity is similar in both ears IAT 410

6. Pinnae Filtering • Outer ear (Pinna) shape filters sound based on its direction • Childhood learning trains brain to associate filtering effects with direction • Unique per person • Record directional white noise • Microphone in ear canal • Sounds from speakers located about head IAT 410

7. Pinnae Filtering • A “Generic” Pinna can be simulated • Record directional white noise received by dummy head • Body filtering • Reflection and absorption • Included in Pinna model IAT 410

8. Head-Related Transfer Function • HRTF is the general term • Transformation of “real” sound to spatialized sound • Best delivered by earphones IAT 410

9. Environmental Effects • Sound exists in an environment • Bounces off objects • Is absorbed by objects • Simple effects • Reverb: Simulate the environmental echo • Echo is the attenuated signal • Gives a richer room-like feeling • Larger room has longer time delay IAT 410

10. Audio signals • Nyquist limit: • Must sample signal at least twice as frequently as highest reproducible frequency • Audio: 44.1KHz (CD) • 22KHz • 11KHz (Analog AM Radio) • 8KHz (Telephone) IAT 410

11. Audio - Digital Implications • 44,100 Hz • 44,100 Samples/sec • 16-bit samples • Stereo • 172KBytes/sec • Specialized hardware - Sound card IAT 410

12. Reproduction • Sampling • Record sounds by whatever means • Synthesis • Analog Synthesis • FM Synthesis • Wavetable Synthesis IAT 410

13. Control • MIDI - Musical Instrument Digital Interface • Developed to control music synthesizers • Details of synthesis are controlled by synthesizer • MIDI data • Sets synthesis parameters • Sets music sequence IAT 410

14. Synthesis • Analog Synthesis • Simple sum of frequencies • Select from a palette of source frequencies • Sum of frequencies is filtered • FM • One frequency is controlled by another • Wavetable • Digitize audio signals IAT 410

15. Analog Synthesis • Fourier’s observation • Any signal can be created as the sum of sine waves • Square wave: Infinite sum • f + 2f + 4f… • Synthesizer: • Collection of oscillators IAT 410

16. Frequency Domain • A perfectly periodic signal plotted in the frequency domain (Time domain plot) IAT 410

17. Spectrum • Spectrum represents the set of frequencies present in the signal IAT 410

18. Filters • Eliminate part of the signal by removing certain frequencies • Analog filters don’t have these “square” response shapes • Band pass • Bandwidth IAT 410

19. FM Synthesis • Modulate the frequency of a wave • Carrier frequency is modulated by Modulator signal IAT 410

20. FM Synthesis for Synthesizers • The greater the Modulator amplitude, the greater the Carrier frequency variation • Higher Carrier bandwidth • DX: Carrier and Modulator are “musically-tuned frequency” • Depends on the note you are playing • Controls the harmonic content of a note IAT 410

21. Wavetable Synthesis • Collect a sample of the real sound • Issues: • Reduce memory load by looping sample • Shift pitch instead of sampling each individual note • Apply interpolation techniques to make pitch shifting work right IAT 410

22. Raw Sound (Sample, FM, etc) Tuned MIDI Note Sequence Resampling Audio Path Reverb, Environment. Spatialization Envelope Loudness Control Mixing/ Combination IAT 410

23. Wavetable Synthesis Example • Leyanda (Guitar) • Leyanda (CDShaw) IAT 410

24. Interactive Sound • Goal • Want to enhance the interactive experience • Give the user a sense of presence • Add to the emotional content of the game • Make it more fun IAT 410

25. Interactive Sound • Music • Sound effects • Noises • Commentary - Sports • Narrative • Conversations IAT 410

26. Interactive Problems • Regular music composition has • Beginning • Middle • End • Interactive user control makes this difficult • Some genres have this structure IAT 410

27. Interactive Music • Game genres with order • Sports • Racing • Fighting • Semi-Ordered • Puzzle • Adventure IAT 410

28. Music Genres • The Infinite Loop • Theme and variation plays forever • Pomp & Circumstance • Diablo • Problems: • 30 second piece repeated over 6 hours! • 720 repetitions! • Diablo example: 12 Repetitions/hour IAT 410

29. Repetition Solutions • Make the Dominant theme hard to find • No catchy theme! • Create a variety of textures • Make only transitions stand out • Where repetition is small • Don’t repeat musical phrases IAT 410

30. Music Strategies • Play Win or Lose music • Music must be long enough to be meaningful • Music may be so long that the game situation changes before completion • Very short music makes little sense • Interrupt current music • Sounds jarring IAT 410

31. Modules • Modular chunks • Each segment of the game plays independently of others • Some thematic relation • Disjointed IAT 410

32. Music Strategies • Compose many themes in parallel • Switch between themes • Connect modular components together IAT 410

33. Analogy: Parallel Trains • N trains of music running in parallel • Each train serves an emotional purpose • Train A: Calm • Train B: Rising Excitement • Train C: Climactic moments • Train D: Falling Excitement • Generally, Train A would be most commonly played IAT 410

34. Within Each Musical Train • Each “Car” contains a few bars of music • Switch between trains when a “Car” is complete • Don’t switch in the middle of a “Car” • Simple version: • Each musical phrase ends on last bar of “Car” • Complex: • Notes at end are carried over to next IAT 410

35. Bars 1-8 Bars 1-8 Bars 1-8 Bars 1-8 Bars 9-16 Bars 9-16 Bars 9-16 Bars 9-16 Bars 17-24 Bars 17-24 Bars 17-24 Bars 17-24 Bars 25-32 Bars 25-32 Bars 25-32 Bars 25-32 Parallel Trains IAT 410

36. Parallel Trains: Shuffle Cars • Shuffle cars • Instead of playing cars in order • Problem: Random cars sound like random radio tuning • Must determine • Appropriate car pairings • Reasonable paths IAT 410

37. Repetition In Trains • Use repeated phrases carefully • Maybe use a statistical tool to analyze paths • Bayesian nets • Endings: • Use transitions as an opportunity to “End” • Use next Car to “Begin” new series IAT 410

38. Composing a Train • Create a piece with all layers • Piece can probably survive a layer or two removed • Variation = piece with layer removed • Be careful with prominent instruments • Fallback: Use instruments with similar acoustical properties • Piano, Organ, Woodwinds • No Trumpets, Drums or Screaming guitar! IAT 410