DSP: SHADERS AND LIGHTING FOR AUDIO

2. DSP: SHADERS AND LIGHTING FOR AUDIO Guy Whitmore Director of Audio Microsoft Game Studios

3. An Apt Analogy If raw SFX are akin to 3D visual models, then runtime audio DSP (Digital Signal Processing) is akin to lighting, pixel shaders, and particle effects.

4. Agenda • Current state of DSP in games • Potential for DSP use in games • DSP middleware in traditional linear audio tools • Creating a DSP middleware market for games

5. Why Run-time DSP? Why not simply bake it in? • Blending and cohesion • Variation • Dimension and depth • Flexibility • Mix iteration and fine control • Shape the mix: Sweeten or fatten

6. Keepin’ It Real • In Linear media, DSP is typically kept 'real' until the final mix stage • In games, final mix is the output of the console as it's being played! • The work of the sound designer continues within the game itself.

7. Common Uses of DSP in Games • Environmental Reverb • Hardware based vs. software • Basic EQ and Low Pass Filters • LPF over distance or underwater • Pitch shifting and dopler • Variation • Car Engines

8. Further Uses for DSP • Radio chat DSP • Early reflections for environmental reverb • USB microphone input • Pitch detection/correction • Time stretch/compression • Aural exciter on swords • Live chat DSP to morph voice

9. Game Parameters Directly Tied to DSP • Time of day, change of seasons • Weapon magic or quality over time • Distortion on highest car RPMs • LPF on unloaded auto RPMs (deceleration)

10. Run-time Mixing and DSP • Mix sets with different DSP settings • Change the mood from one scene to another • EQ and compression intended for mixing via groups • Channel strips for character VO • Mastering chain DSP • Mastering EQ, compression, limiting

11. Where Does DSP for Games Currently Come From? • As part of Console SDKs • Reverbs, compression, EQ • As part of an Audio Engine license • Coded in-house • Centralize your custom DSP Result: Limited range and variety of DSP

12. DSP in Traditional Linear Audio Production Digital Audio Workstations (DAWs) • Come with many DSP plug-ins • Plug-in archetechture enables 3rd party DSP makers to participate in a DAW's ecosystem. Result: • Large and varied 3rd party selection of DSP • High end brand name to free-ware

13. Why so Many DSP Makers? Isn’t a reverb a reverb? • Choice • Aesthetics and taste • Variety • Differentiation • Currently most 360 titles use the XDK reverb • Appropriateness to the title • CPU and tech requirments vary

14. DSP Licensing For Games…so far • Console makers/card makers • Xbox 360 – Princeton Digital Reverb • Xbox (and DirectMusic) – Waves Reverb • Audio Engine companies? • Wwise - McDSP • Developer licenses • Bungie - Waves • Publisher licenses • MGS - Wave Arts Ltd.

15. MGS and Wave ArtsA pattern for future deals In short, it’s a… • multi-license • multi-algorithm • Master Verb, Trackplug, Final Plug,Multidynamics • multi-platform • Xbox 360 and PC • Multi-format deal • XAPO, generic, • Wwise and FMOD plug-in compatibility coming • Structured to give Wave Arts the upfront capital to port the plug-ins.

16. Wave Arts Now that the upfront work is done, other publishers and developers can directly license DSP from Wave Arts. Possibilities: • More algorithms  • Port to PS3? …to Wii?

17. The DreamWouldn’t it be cool if… • The extensive variety of DSP brands used our linear DAWs were available to us in our game environments? • We could easily 'plug-and-play' new DSP in our audio engines (no coders) for seamless auditioning? • Licencing 3rd party DSP was the norm, relatively easy, and affordable?

18. A 3rd Party DSP Market for Game Audio Benefits: • Access to a wide range of quality algorithms • Independent of game platforms and audio engines • Easy to audition and edit in game and offline • Sound designers are familiar with many of the brands and algorithms

19. Technical ChallengesThe Easy Part (relatively speaking) • Porting DSP to various game platforms • Optimizing DSP to console archetechture and for efficiency. • Availability of dev kits for DSP makers • Various plug-in formats • An industry standard plug-in format is a worthy goal and should be pursued, but it’s not mandatory for a successful middleware ecosystem.

20. Financial ChallengesWhat will the market bare? • How should DSP licenses be charged? • What's a fair price per SKU? • Affordable by developers/publishers • While making the effort worthwhile for a DSP maker • Differentiation between DVD and downloadable games? • Audio/game budgets for DSP licenses

21. Chicken Egg ConundrumHow to kick-start a middleware ecosystem Kindling for the fire • Build a critical mass of licensors • 3-5 deals similar to Wave Arts

22. An Industry ChallengeTo major game publishers and audio leaders: Strike new DSP porting/licensing deals with existing DSP makers • Modeled similarly to the MGS – Wave Arts deal i.e. • Multi-license and multi-algorithm • DSP company able to license algorithms to other developers after contract term • Work with DSP companies that… • Make interesting quality algorithms • Show willingness to forge into a new market • Create CPU efficient algorithms • Will support their products into the future

23. ??Questions?? ?????????????? ??????? ??????????