Introduction to Game Audio Why Film is Easy and Games are Hard

1. Introduction to Game AudioWhy Film is Easy and Games are Hard Brian Schmidt President, Brian Schmidt Studios Executive Director, GameSoundCon

2. History of Game Audio topinit lda #$4f top deca out a,#$7ffe cmpa #$0 timrloop ldb $#timr cmpb $#8000 bne timrloop bne top bra topinit track1 patch Bass33 volume 43 loop 8 note c1,30,20 note c2,30,28 endloop endtrack track2 loop patch kick note c2,60,10 patch snare note c3,60,15 endloop endtrack • Early-early days • Raw computer code • All those ‘video game’ sounds (Defender, Robotron, etc.) • Getting better (Genesis, SNES) • Tiny synthesizers (6-8 voice) • MIDI-like engines • But not MIDI (MIDI was pretty young) • Hand-coded • Very limited memory (64k!) • Very limited synthesis FM or simple sample

3. CD Era • Playstation 1 • CD • “Redbook audio” • Lots of room for music, right? • Lots of room used for other stuff

4. 2000 DVD: The Storage Explosion..Playstation 2, Xbox • Use of DVD as game medium disruptively changed game audio • Multiple streams • Consumer expectations (DVD = high quality, high fidelity) • Gigabytes of content • Music and dialog! • Tens of thousands of lines of dialog • “goodbye real-time generated music…”

5. Current Consoles • Xbox 360 • Hundreds of sounds, fx, surround • Software & hardware audio • DVD • Playstation 3 • Extremely powerful software audio • Hundreds of sounds, fx, surround • Bluray • Wii • Generally more limited • MIDI + studio-produced music

6. Game Sound has Come a Long Way…. • Technology has taken care of many issues • High fidelity Synthesis and playback • 48kHz, Surround Sound, Signal Processing • Streaming • Live orchestras, Complex ambiences • Effects Processing • Modern game consoles have “ProTools” levels of processing power

7. Film sound is straightforward—Game sound is not So why are films easy and games hard?

8. Challenges of Game Audio

9. Game Audio Challenges • Challenges affect each other • Will vary greatly from game to game • Concepts important

10. Aesthetic and Creative Challenges • No Precognition • Duration of experience • Sound reacts with “physical” world • Nature of sounds can depend on context

11. Linear Media ‘Cue Spotting’

12. Video Game– no “spotting”

13. No Precognition • Specific game events, actions, are not known in advance • User actions dictate flow and timing • User responses initiate new actions • Impossible to map out entire flow • Almost limitless possibilities

14. How do Games deal with lack of Precognition? • “Event mapping” rather than spotting • Determine all possible actions that would require sound or change in sound state • Often called “Triggers,” “Cues” or “Events” • Create a “sound” for each event

15. Minor Digression: What is a “Sound” • “Anyone who believes there is a one to one correspondence between a game event and a wave file doesn’t know game sound.”* • Buzz Burrows (or Brian Schmidt, depending on who you ask)

16. How do Games deal with lack of Precognition? • Flexible sounds • Sounds with parameters • Not just “impact” but “impact” with “how hard” • Flexible music • Pre-composed ‘chunks’ that can relatively quickly adjust to games’ mood • “arranged on the fly” • Dialog for various emotional state

17. Duration of Experience • “First, do no annoying” • Marty O’Donnell, HALO composer & Audio Director • Most console games are long • 30,40,50 hours of gameplay • Ear-fatigue is issue • Even the best music can get tiresome • Repetition destroys immersion • Variation is essential

18. How do Games deal with Duration • Don’t need music all the time • Experiments with Generative Music • Lots of ‘redundant’ assets • Don’t keep the best take– keep the best 20 takes • Get the most out of assets • Pitch shifting, subtle filtering • Keep some surprises for the die-hards • Eg a dialog phrase that’s only heard rarely

19. Good use of dialog Dialog Selection 1% of time 99% of time 50 dialog lines 10 “rarely heard” lines

20. Sound Interaction • Sounds interact with the world • And need to sound different based on context in which they’re played • Footsteps on a highway--- or in a cavern • Voices may be through “comm” system • Radioized..or not • Could change in middle of line • Impractical/Impossible to create individual wave for every possible location/circumstance

21. So, effects best applied at run-time, not at production time • Impossible to do with ProTools plugins • Hot topic in game sound…

22. Technical Challenges

23. Why do we care about all this techie stuff? • Games are a “shared resource” system • Music doesn’t exist in isolation • Shared resources affect one another • Technology enables (and restricts) what can and can’t be done • Knowing the technology allows you to make best use of available resources • Arms you to fightwork with the programmer…

24. Goals: • Maximize audio quality while minimizing impact on the other elements of the game • Knowledge of how the system works is essential to that goal • Maximize your own value • More you know, less replaceable you are • Put yourself into leadership/vision position • Extremely difficult without more depth of knowledge

25. What Else is Vying for Game Resources • Graphics • Physics • “Artificial Intelligence” (AI) • Input • Networking • Voice • …

26. Inside a Video Game

27. Game Engine • Controls the universe • Controls graphics, audio, AI, physics • Reads user actions (controller, etc.) • Affects the state of world • Sends commands to the audio engine • Plays sounds, dialog, etc • Changes position of sounds • Initiates changes in music • Other housekeeping tasks

28. Audio Engine • Audio engine takes commands from the game, decides what to play and plays it • Performs real-time synthesis and mixing of all game sound items • Monitors game data necessary to affect sound at the moment • Eg panning a flying helicopter, changing sound of a football crowd based on emotional crowd state, etc

29. May be random May be random Game Event Determine which wave(s) to play Determine Parameters for Playback Play May be based on game parameter(s) May be based on game parameter(s) (Doppler, 3D position, etc.)

30. Technical Challenges • Resources are shared • Graphics, Networking, Physics, AI, Audio all vie for system resources such as memory, CPU, storage • Technical limitations • Memory • CPU • Data transfer & bandwidth • Many systems “roll your own” • What you did on one game may be totally different on the next • Integration of audio assets into the game not at all straightforward

31. Additional Technical Challenges • New definition of “Mixing” • Tools for game audio are in early generations • No/limited standards • Lots of custom, proprietary solutions • Lots more on this topic in main GDC sessions

32. Audio Development Mechanics • All over the map • Console games require special “dev kits” to run prototype games • Available only to “licensed developers” • Allow you to take a ‘build’ and play it • Allow you to develop sounds and hear on actual hardware

33. Development Issues • Possible to prototype on PC • Game Tools generally Windows based • Best to at least install Parallels or Bootcamp if you’re doing game work • Or get small PC for game audio use • You’ll likely be told which tool to use by game developer • “just wave files” • Custom in-house tool • 3rd party Middleware

34. What we Learned • Game Development presents unique audio challenges • Lack of precognition • Duration of Experience • Sounds that vary with action • Resource constraints • Overview of a video game • Need to know multiple toolsets • Or concepts so can learn custom ones

35. Organizational Challenges • Game Production is software development • Testing • Asset Count • Tens of thousands of lines of dialog • Shared resources • Audio on the radar too late • No “Post production phase” • Remote worksite • Audio programmer generally low man on totem pole

36. Game Team • Designer(s) • Producer(s) • Programmers (dozen or more) • Artists (10,20,60..or more) • Testers (several) • Audio Team (1-4)

37. Basic Game Development Cycle Repeat Daily for 2 years

38. Build Process Usually Overnight

39. Sounds Straightforward • Games have hundreds of thousands of individual files • .wavs, art files, code files, script files, etc. • Easy for something to go amiss • Some files depend on other files • Change in a filename • Typos • Result is a “Broken Build” • Game can’t be built or played

40. The Players • Console Manufacturers • Sony, Microsoft, Nintendo (plus smaller ones) • Game Publishers • Electronic Arts, Activision, Sony, Microsoft, Nintendo, Ubisoft • Game Developers • Red Octane, High Voltage, NuFX….. • Contractors

41. Console Manufacturer • Designs and makes consoles • Makes software for game developers to make their games • The “SDK” (Software Development Kit) • Makes special hardware for game developers • Also publish games • Also develop games • Often publish and develop most profitable titles for their console • Halo/ Mario, etc

42. Game Publishers • Fund Game • Market Game • Sell Game • Relationship with retailer • Use internal or external developers • Maintain relationship with console manufacturer • Electronic Arts, Activision, Warner Brothers, etc

43. Game Developer Usually this is who you work for • Often (but not always) designs game • Creates code, art, sound for game • Usually (but not always) funded by publisher • Generally responsible for hiring audio • In-house or free-lance

44. “Indy” development • Recently, self-publishing on major platforms is possible • Xbox “XNA Game Studio”, “Wiiware,” iPhone and other “app stores” • Cut out the publisher • Analogous to indy labels • Opportunities abound • $$ not so much • Great opportunity to cut your teeth

45. Who do you work for? • In-house • Developer or publisher • Generally “audio director” jobs • Some composition, but typically as part of a larger capacity (vision & direction, etc) • Free-lance • Most often contracted with Developer • Occasionally contracted by Publisher • Varies with publisher/developer

46. Last note on business • Game Development companies are only slooowly learning about music publishing and rights • Standard contract is “Work for Hire” • Industry in transition • Make sure your lawyer is familiar with game industry

47. What we Learned • Game Development is very different from linear media • No shortage of creative, technical and process challenges • Game creation is a software development process as much as (or more so than) creative process • Several players • Console maker, publisher, developer, free-lance

48. Opportunities: Careers • Composer/Sound Designer • Audio Integrator/Technical Sound Designer • Audio Director • Audio Programmer • Localization • Dialog • Everything • Esp for small scale/Indy games

49. Opportunities: Gigs