Interactive Audio Design: Where the Heck do we go from Here??

1. Interactive Audio Design:Where the Heck do we go from Here?? Brian Schmidt A software company

2. Premises • We are at the most exciting time: • The creation and evolution of a new media and art form: Interactive Audio Design and Delivery • The issues are still fuzzy, but getting clearer by the day • Plenty of BIG issues still need to be solved

3. Our Holy Grail The creation and delivery audio soundtracks for interactive entertainment that give the illusion that they were created post-production, but are generated or assembled at run-time.

4. Our Holy Grail (paraphrased) Make games stop sounding like $%%&$@ games!!!

5. Our Holy Grail:How we find it(where the heck do we go) • Identify challenges • Look at people, roles, aesthetic issues • Identify tools, technologies, to be addressed

6. Challenges Above and Beyond Linear Entertainment • Music and sound effects need to FIT. • Not “kinda fit,” “Sorta go with,” “almost get it,” “be generally the right mood…” • Music and sound need to be as tightly integrated to action as linear media, without having the advantage of precognition • Delivery platforms vary • Leads to “least common denominator” implementations

7. Challenges Above and Beyond Linear Entertainment • Difficult to harness nuance and musicianship of artists and performers and have soundtrack fit with media • Pre-scored music captures nuance at expense of flexibility • Musicians perform on non-traditional instruments, if at all • Too much reliance on keyboard as input medium

8. Challenges Above and Beyond Linear Entertainment • All media decisions in an interactive product interact with and have ramifications with the rest of the product • Most obvious: Processor/Frame rate concerns • Also other resource usage issues (cd, dvd, memory etc.) • Audio Architect’s role is to scope the project to maximize audio in context of game

9. New paradigms mean expanded Job Descriptions • Specialization • Composer • Interactive arranger • Sound Designer • Audio Integration Engineer • Audio Architect • Describes tasks, not necessarily headcount • Headcount comes with maturity of industry

10. Audio Job Categories:Composer • Writes da music • Probably orchestrates the music • Creates musical themes, building blocks • Works within constraints provided by Audio Architect

11. Audio Job Categories: Interactive Arranger • Analogous to arranger/orchestrator for Film music • Formats composed pieces for integration into game • Works closely with rest of game team

12. Audio Job Categories:Sound Effects Designer • Identifies required sound effects for title • Leads “spotting” sessions • Creates sound elements • Works within constraints provided by Audio Architect • Provides direction to Audio Implementation Engineer

13. Audio Job Categories:Audio Implementation Engineer (AIE) • Responsible for integrating audio elements into game • Integrating = “programming” • Works with programmer to implement audio, spot queue points, etc. • Analogous to “Audio Engineer” in tradition post-production • Works closely with Interactive Arranger and Sound Designer. • Helps arranger/sound designer utilize technology to the fullest

14. Audio Job Categories:Audio Architect • Scopes and defines suitable audio implementation strategies with “the big picture” in mind • Works closely with rest of game and audio team • Identify resource issues • Tracks and estimates performance implications • On the project from Day Zero • Provides blueprint for audio implementation that works within the entire game • Maximizes audio without negatively impacting game as a whole

15. Aesthetic issues • As long as sounds are just “wave files”; as long as music is “fire and forget,” games will continue to sound like “games.”

16. Aesthetic Issues • Compositional Techniques • Some borrowed from film? • Foreshadowing • Musical “fake out” • Some New • How interactive does it need to be • Composing non-linear music that still has form • Use of performers for real-time rendered scores

17. Aesthetic Issues • Designing sound effects designed to be played/listened to multiple times • Avoiding predictability and monotony • “Sounds for every situation” • The above weighed against the need to provide information • Far more important than for linear media

18. Technology: Major Areas for Advancement(What we should be working on) • Production Tools and Technology • Delivery Technology

19. Production Tools and Technologies • Non-linear sequencers • Allow composers to write music designed to be played back interactively • Deep musical representations • Provides more “knobs” on musical control • Mood, key, intensity, etc. • Tools to author to specific run-time rendered environments • Musically relevant sound generation tools • Audio designer-friendly access to rendering and playback parameters

20. Inter-Production Tools and Technologies (2) • Audio integration environment • Perhaps the #1 most important issue keeping audio quality down • Analogous to the post-production studio • Audio Integration and contextual auditioning must be independent from rest of game development process • Economics and production realities demand it • Development Tools that do not require significant interaction with the rest of the development process

21. Create Media Give to Programmer WAIT Listen in Context

22. That process is broken

23. What’s Broken? • Too much reliance on programmer • Turn around far too long • “It’s easiest to implement” is often an implementation criteria • Weak link • Better question “What’s not broken?”

24. Process must be more similar to linear media “post production” • Audio integration must be able to be performed by audio experts • Can not rely on re-compile, etc. for “edit, listen, modify” loop • Technological issues must be well understood by those creating the content

25. Delivery Technologies Required • Extraordinarily high quality run-time rendered music and sound effects • “Studio in the box” • Sophisticated synthesis technologies • High level delivery technologies • Seamless integration of pre-rendered and run-time rendered audio • Multi-faceted audio delivery systems • Provides different ways to provide audio for specific situations

26. Summary • We are 150 miles into a 1000 mile journey • Non-trivial progress but a long way to go • Lots of great technologies to invent and deliver • Lots of great creative challenges ahead • This room has the best collection of people on the planet to tackle these issues.