1 / 55

Compelling audio and video for Metro style games

PLAT-755T. Compelling audio and video for Metro style games. Jason Strayer Senior Program Manager Microsoft Corporation. Agenda. Audio and video features that make a compelling game Features your users will expect to see and hear and features that will make your game shine!

finn
Download Presentation

Compelling audio and video for Metro style games

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PLAT-755T Compelling audio and video for Metro style games Jason Strayer Senior Program Manager Microsoft Corporation

  2. Agenda • Audio and video features that make a compelling game • Features your users will expect to see and hear • and features that will make your game shine! You’ll leave with examples of how to • Implement common audio features such as sound effects • Render video in your game to Direct3D textures

  3. Audio

  4. Audio in your game

  5. Game audio features Audio • Sound effects • Background music • Dialogue • UI sounds • Voice chat • 3D positioning • Environmental audio • Multi-channel output

  6. demo Simple3DGame Demonstrates audio usage in a simple DirectX 3D-based game

  7. Sound effects • Triggered and synchronized with game behavior • requires low latency playback • and sample-accurate looping control • Many concurrent sounds playing at same time • requires robust sound mixing of multiple concurrent voices • Variations in audio needed to maintain sense of realism • requires synchronous sample accurate playback • and dynamically manipulated sound buffers

  8. API overview: XAudio2 • Low latency sound mixing engine • Designed specifically for game scenarios…but useful in many types of real-time audio apps • Built as a cross-platform Xbox/Windows audio API • Now ships as part of Windows 8 • Headers and libraries ship in Windows SDK for Metro style apps

  9. Audio in the game world Enemies Alarm Explosion Weapon fire Teammates Player

  10. XAudio2 audio graph architecture Nearby Alarm Obstruction submix Final mix to output stream Player’s Weapon Effect Chain Source Voice Memory Buffer(s) Enemies Behind Box Source Voice Memory Buffer(s) WASAPI Output Mastering Voice Effect Chain Effect Chain Source Voice Memory Buffer(s) Effect Chain Submix Voice Memory Buffer(s) Source Voice

  11. Basic sound effect playback • //Create the XAudio2 engine and mastering voice on the default audio device • XAudio2Create(&m_audioEngine); • m_audioEngine->CreateMasteringVoice(&m_masteringVoice);// Load all audio data for the sound effect into a single in-memory buffer • MediaLoadersoundFile(ref new Platform::String(SOUND_FILE));m_soundEffectBuffer = soundFile.ReadAll();// Create a single source voice for a sound effect • m_audioEngine->CreateSourceVoice(&m_sourceVoice, &(soundFile.GetOutputWaveFormatEx())); • //Trigger sound effect: queue in-memory buffer for playback and start the voiceXAUDIO2_BUFFERbuf = {0};buf.AudioBytes = m_soundEffectBuffer.size();buf.pAudioData = &m_soundEffectBuffer[0];buf.Flags = XAUDIO2_END_OF_STREAM;m_sourceVoice->SubmitSourceBuffer(&buf);m_sourceVoice->Start();

  12. demo Marble Maze Streaming music with media foundation in a DirectX game

  13. Background music • Indefinitely streaming ambient music during gameplay • requires efficient run-time storage and loading of assets • and requires audio compression to reduce storage and memory • and may require protected content • Overall system needs to manage multiple streams • requires system level muting and ducking • which requires stream audio category tagging

  14. API overview: media foundation (MF) • End-to-end extensible media solution • Audio/Video playback & capture • Media foundation transforms (MFT) and codecs • Source readers and sink writers • Win32 C++ API available for DirectX C++ Metro style apps • Supports low latency option for real time performance

  15. Windows Runtime media platform Windows Runtime (WinRT) Playback/Preview Capture Transcode MediaControl Streaming Extensibility Protection Media Foundation Audio/Video Source Video Decoder Video Effect 1 Video Effect 2 Video Encoder Audio/Video Sink Audio Decoder Audio Effect 1 Audio Effect 2 Audio Encoder Direct3D Windows Audio Session API (WASAPI)

  16. Windows Runtime media platform Windows Runtime (WinRT) Capture Transcode Streaming Extensibility Protection MediaControl Playback/Preview Media Foundation Audio/Video Source Video Decoder Video Effect 1 Video Effect 2 Video Encoder Audio/Video Sink Audio Decoder Audio Effect 1 Audio Effect 2 Audio Encoder Direct3D Windows Audio Session API (WASAPI)

  17. Audio playback using media engine • <audio>, <video>, XAML MediaElement from C++ • Media engine encapsulates media foundation components • Relatively minimal code needed to add to your game • Provides basic playback controls • Supports real-time mode for reduced latency • Expected latency in 100ms range

  18. Media engine architecture Media Engine WASAPI Output Media Source Source File Audio Decoder Media Sink

  19. Media engine audio playback • Microsoft::WRL::ComPtr<IMFMediaEngineClassFactory> mediaEngineFactory; • Microsoft::WRL::ComPtr<IMFAttributes> engineAttributes; • Microsoft::WRL::ComPtr<IMFMediaEngineExtension> pExtension; • MFStartup(MF_VERSION); • // The Media Engine requires a registered callback handler • m_engineNotify= new (std::nothrow) MediaEngineNotify(); • // Create the class factory for the Media Engine. • CoCreateInstance(CLSID_MFMediaEngineClassFactory, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&mediaEngineFactory)); • // Set configuration attributes, including audio category • MFCreateAttributes(&engineAttributes, 1); • engineAttributes->SetUnknown(MF_MEDIA_ENGINE_CALLBACK, m_engineNotify.Get()); • engineAttributes->SetUINT32(MF_MEDIA_ENGINE_AUDIO_CATEGORY, AudioCategory_GameMedia); • // Create Media Engine with real-time mode • constDWORD flags = MF_MEDIA_ENGINE_AUDIOONLY | MF_MEDIA_ENGINE_REAL_TIME_MODE; • mediaEngineFactory->CreateInstance(flags, engineAttributes.Get(), &m_mediaEngine); • m_mediaEngine.As(&m_mediaEngineEx); • // Set the location of the media source then begin streaming • BSTR bstrURL = SysAllocString(pszURL); • m_mediaEngine->SetSource(bstrURL)); • m_mediaEngine->Play();

  20. Media engine architecture Media Engine WASAPI Output Media Source Source File Audio Decoder Media Sink Source Voice Memory Buffer(s) Mastering Voice Effect Chain Effect Chain Source Voice Memory Buffer(s) Effect Chain Submix Voice Memory Buffer(s) Source Voice

  21. Audio compression: size vs. speed • PCM data is the lingua franca of audio APIs • WAVE files are the most rudimentary container format • Shipping all of your assets as PCM-encoded WAVs • Might give you the best performance + quality, but... • It will bloat your download size and memory footprint dramatically • Solution: you must compress! • Encode one-shot sound effects as ADPCM using ADPCMENCODE • Encode very long sounds (dialogue, music) as WMA / MP3 / AAC

  22. Audio compression formats ADPCM PCM

  23. More on decoding • Media foundation uses hardware offload when available • Hardware offload becoming common feature in tablets today • Advantage of using media engine for your background music • Use sparingly: hardware designed for minimal concurrent streams • Source reader can be used to stream+decode to PCM buffers • Allows use in XAudio2 for additional effects or mixing • Use for AAC/WMA/MP3 content

  24. 3D positioned sound sources Emitters Alarm Emitters Weapon fire Emitters Emitters Listener

  25. 3D positioned sound sources • Use XAudio2 X3DAudio functions for mathematical support • Simple one-function use: X3DAudioCalculate per emitter/listener pair • Objects in world act as “emitters” • Player (or camera) acts as “listener” • Maps easily onto 3D objects in your scene

  26. Environmental effects Reverb Occlusion Obstruction Player

  27. Environmental effects • Apply reverb and echo effects in appropriate environments • Use low pass filter for occlusion and obstruction • Implemented through software DSPs (“APOs” in XAudio2) • XAudio2 provides built-in effects that are ready to use • Two reverbs, echo, volume limiter, equalizer, volume meter • Differentiate by creating your own unique effects • Or license from available professional solutions

  28. Audio device disconnects • Audio devices can be disconnected at run-time • Usually requires resetting state of audio engine • Due to change in number of channels, output format or bitrate • XAudio2: Handle IXAudio2VoiceCallback::OnCriticalError • Destroy and release all voices, recreate graph • MediaEngine: automatically transitions to next audio device

  29. Video

  30. Game video features Video • Pre-rendered cut-scenes • In-game video • Video textures • Video enhanced effects

  31. demo TabletSim DirectX 11 video to a Direct3D texture

  32. Pre-rendered cut-scenes • Use cut-scenes to enhance game experience with extra context • Transition between levels or game segments • Provide game objectives viscerally • World background, character history • Great reward for player accomplishments • Can be easier to display video than render using game engine • Allows level of fidelity much greater than basic rendering • Typically played at full-screen and interruptible

  33. In-game video • In-world video elements add real-world character • Television screens, monitor display • Advertising, video billboards • Allows game to enhance video with game elements • Static effects, damaged screens • Avatar audience reacts to video as it plays • Typically rendered directly to Direct3D surface • Allows for simulated texture animation on models

  34. API overview: DirectX 11 video • DirectX 11 Video accelerates video decode in MF pipeline • Rendering video to textures is simple • Use MF Source Reader to decode from a video source • For video MF samples provide D3D11Texture* resources • Use directly as texture for rendering • Keep your frame-rate up! • Underlying video codecs can use GPU decode • Recommended formats: WMV, H.264 • DXGI adds new YUV formatsfor use as output from codec for use in rendering pipeline with new BIND semantics

  35. Create the DXGI device manager • MFCreateDXGIDeviceManager(&m_resetToken, &m_d3dDeviceManager); • IUnknown* pUnk;m_d3dDevice.CopyTo(&pUnk); • m_d3dDeviceManager->ResetDevice(pUnk, m_resetToken); • Microsoft::WRL::ComPtr<ID3D11DeviceContext1> immediateContext; • m_d3dDevice->GetImmediateContext1(immediateContext.GetAddressOf()); • Microsoft::WRL::ComPtr<ID3D10Multithread> d3dMT; • immediateContext.As(&d3dMT); • d3dMT->SetMultithreadProtected(TRUE); • MFCreateAttributes(&mediaAttributes, 2); • IUnknown* pUnk; • d3dDeviceManger.CopyTo(&pUnk); • mediaAttributes->SetUnknown(MF_SOURCE_READER_D3D_MANAGER, pUnk); • //Create Media Foundation Source Reader to use for decoding into in-memory buffersMFStartup(MF_VERSION);MFCreateSourceReaderFromURL(URL->Data(), mediaAttributes.Get(), &m_sourceReader);

  36. Set output media type • Microsoft::WRL::ComPtr<IMFMediaType> mediaType; • // Illustrating simple usage - set the decoded output format as RGB32MFCreateMediaType(&mediaType));m_sourceReader->GetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, mediaType.GetAddressOf())); • pMediaType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video)); • pMediaType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_RGB32)); • m_sourceReader->SetCurrentMediaType(MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, mediaType.Get()));

  37. Decode next video frame • Microsoft::WRL::ComPtr<IMFMediaBuffer> mediaBuffer; • Microsoft::WRL::ComPtr<IMFDXGIBuffer> DXGIBuffer; • LPVOID resource; • DWORD flags = 0; • // Preserve MFSample object until video texture has been used for rendering • // Timestamp should be used in render loop to time frame output • m_sourceReader->ReadSample(MF_SOURCE_READER_FIRST_VIDEO_STREAM, 0, NULL, &flags, • (LONGLONG*)timestamp, &m_mediaSample); • // Retrieve texture object from sample • m_mediaSample->GetBufferByIndex(0, &mediaBuffer); • mediaBuffer.As(&pDXGIBuffer); • DXGIBuffer->GetResource(__uuidof(ID3D11Texture2D), &resource); • *videoTexture= reinterpret_cast<ID3D11Texture2D*>(resource); • // Use videoTexture as input texture in Direct3D render pipeline

  38. Review

  39. Media platform for DirectX apps Game Engine Windows Runtime (WinRT) XAudio2 Media Foundation Audio/Video Sink Audio/Video Source Video Encoder Video Effects Video Decoder Audio Decoder Audio Effects Audio Encoder Direct3D Windows Audio Session API (WASAPI)

  40. Great audio and video in entertainment appsis no longer a differentiator,it is a requirement.

  41. Windows has the functionality you needto create compelling audio and video experiencesfor your players.

  42. Related sessions • [PLAT-775T] Your Metro style app, video and audio, Part 1 • [PLAT-776T] Your Metro style app, video and audio, Part 2 • [PLAT-778T] Media fundamentals of a communications app • [PLAT-783T] Extending the media platform using input, output and processing plug-ins

  43. Further reading and documentation • Developing Games • Developing the Metro Style Marble Maze game with DirectX • Working with Audio in your DirectX game • XAudio2 documentation • Past XAudio2 presentations at Gamefest • Highly recommended: XAudio2: High Performance Considerations • DirectX forum on http://forums.dev.windows.com

  44. thank you Feedback and questions http://forums.dev.windows.com Session feedbackhttp://bldw.in/SessionFeedback

  45. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

  46. XAudio2 Changes for Windows 8 • Inbox component, no longer redistributable • Will include ARM support • Down-level Desktop apps can continue to use DX SDK version • XAUDIO2_8.DLL includes X3DAUDIO and XAPOFX • Removed support for xWMA decoding • Use Media Foundation Source Reader as decoder • No Debug version of DLL • Evaluating runtime validation and ETW traces in future release

  47. Mastering Voice changes for Windows 8 • Device enumeration removed from XAudio2 • Use Windows 8 device enumeration functions instead • CreateMasteringVoice uses device interface ID string instead of index • Developer Preview release only supports default audio render device • IMasteringVoice::GetChannelMask: get mask after device creation • Support for Playback Manager (PBM) categories • CreateMasteringVoice has new category parameter • Optional – defaults to “GameEffect” category

  48. Decoding Buffers for XAudio2 MF Source Reader WMA File MF Source Reader MP3 File MF Source Reader Custom Format Custom Codec ADPCM in WAV File Basic File I/O

  49. Using Media Foundation to Decode, Setup • Microsoft::WRL::ComPtr<IMFMediaType> outputMediaType; • Microsoft::WRL::ComPtr<IMFMediaType> mediaType; • // Create Media Foundation Source Reader to use for decoding into in-memory buffersMFStartup(MF_VERSION);MFCreateSourceReaderFromURL(url->Data(), nullptr, &m_reader);// Set the decoded output format as PCMMFCreateMediaType(&mediaType);mediaType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Audio);mediaType->SetGUID(MF_MT_SUBTYPE, MFAudioFormat_PCM);m_reader->SetCurrentMediaType(MF_SOURCE_READER_FIRST_AUDIO_STREAM, 0, mediaType.Get());m_reader->GetCurrentMediaType(MF_SOURCE_READER_FIRST_AUDIO_STREAM, &outputMediaType);//Obtain WAVEFORMATEX and maximum size of string based on durationMFCreateWaveFormatExFromMFMediaType(outputMediaType.Get(), &waveFormat, &formatByteCount);m_reader->GetPresentationAttribute(MF_SOURCE_READER_MEDIASOURCE, MF_PD_DURATION, &var);m_maxStreamLengthInBytes = (uint32)(var.uhVal.QuadPart/ (float64)(10000 * 1000)) *m_waveFormat.nAvgBytesPerSec;

More Related