1 / 38

Media extensions

Media extensions. Marcin Stankiewicz and Matthew Howard Development - Multimedia Platform Team Richard Fricks Windows Scenario Adoption Team Session 4-104.

luana
Download Presentation

Media extensions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Media extensions Marcin Stankiewicz and Matthew Howard Development - Multimedia Platform TeamRichard FricksWindows Scenario Adoption TeamSession 4-104

  2. Engaging with rich media—whether watching a movie, video chatting, or playing music—is one of the most prevalent and enjoyable things we do on our PCs today.

  3. Agenda • Architecture overview • Media Foundation pipeline • Media extensions • Visual Studio 2012 • Media source demo • Streaming media to JavaScript

  4. Media Application Architecture Windows Store app • Media namespace • Capture • Transcode • PlayTo • PlaybackManager • MediaExtensionManager JavaScript – HTML5 Video tag, audio tag C# MediaElement Windows Runtime (WinRT) Media engine Playback and preview Capture engine Capture Transcode Sharing engine PlayTo Media Foundation

  5. Media app architecture Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Media source Decoder Effect Encoder Sink Audio Audio encoder Audio sink Audio decoder Audio effect Media engine Playback and preview Capture engine Capture Transcode Sharing engine Play To Media Foundation

  6. Media app architecture Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Media source Decoder Effect Encoder Sink • Can be either native C++ DLL or a native C++/CX DLL • The DLL must ship in your app package • You can call WinRT APIs but you can use ONLY those desktop APIs that are marked for Windows Store apps: Audio Audio encoder Audio sink Audio decoder Audio effect http://msdn.microsoft.com/en-us/library/windows/apps/hh452756.aspx

  7. Registration process .RegisterAudioDecoder() .RegisterVideoDecoder() Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline .RegisterAudioEncoder() .RegisterVideoEncoder() Windows.Media.MediaExtensionManager Audio Audio encoder Audio sink Audio decoder Audio effect .RegisterSchemeHandler() .RegisterByteStreamHandler() ?

  8. Registration process: Effects Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Effects are registered using the class your code uses to process the media: Audio Audio encoder Audio sink Audio decoder Audio effect

  9. Registration process: Sinks Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Sinks are supported only by MediaCapture: Audio Audio encoder Audio sink Audio decoder Audio effect

  10. Media source Registration Process: Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Audio .RegisterAudioDecoder() .RegisterVideoDecoder() .RegisterAudioEncoder() .RegisterVideoEncoder() .RegisterSchemeHandler() .RegisterByteStreamHandler() .RegisterAudioDecoder() .RegisterVideoDecoder() .RegisterAudioEncoder() .RegisterVideoEncoder() .RegisterSchemeHandler() .RegisterByteStreamHandler() Audio encoder Audio sink Audio decoder Audio effect Windows.Media.MediaExtensionManager ?

  11. Which to use? • ByteStreamHandler • New container format (WebM, MKV) • SchemeHandler • Generating your own streams • Media stored in a location not accessible to inbox ByteStream • Custom streaming protocol

  12. IMFByteStreamHandlerInterface

  13. IMFByteStreamHandlerInterface • HRESULT BeginCreateObject( [in] IMFByteStream *pByteStream, • [in] LPCWSTR pwszURL, • [in] DWORD dwFlags, • [in] IPropertyStore *pProps, • [out] IUnknown **ppIUnknownCancelCookie, • [in] IMFAsyncCallback *pCallback, • [in] IUnknown *punkState ); MediaElement.Source= newUri(“video.mv4");

  14. IMFByteStreamHandlerInterface • HRESULT EndCreateObject( [in] IMFAsyncResult *pResult, • [out] MF_OBJECT_TYPE *pObjectType, • [out] IUnknown **ppObject ); typedef enum MF_OBJECT_TYPE { MF_OBJECT_MEDIASOURCE, MF_OBJECT_BYTESTREAM, MF_OBJECT_INVALID } MF_OBJECT_TYPE;

  15. SchemeHandlers and Byte-StreamHandlers Byte-StreamHandler Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Audio Audio encoder Audio sink Audio decoder Audio effect

  16. IMFSchemeHandlerInterface

  17. IMFSchemeHandlerInterface • HRESULT BeginCreateObject( [in] LPCWSTR pwszURL, • [in] DWORD dwFlags, • [in] IPropertyStore *pProps, • [out] IUnknown **ppIUnknownCancelCookie, • [in] IMFAsyncCallback *pCallback, • [in] IUnknown *punkState ); MediaElement.Source= newUri("myscheme://square");

  18. IMFSchemeHandlerInterface • HRESULT EndCreateObject( [in] IMFAsyncResult *pResult, • [out] MF_OBJECT_TYPE *pObjectType, • [out] IUnknown **ppObject ); typedef enum MF_OBJECT_TYPE { MF_OBJECT_MEDIASOURCE, MF_OBJECT_BYTESTREAM, MF_OBJECT_INVALID } MF_OBJECT_TYPE;

  19. SchemeHandlers and Byte-StreamHandlers Byte-StreamHandler Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Audio Audio encoder Audio sink Audio decoder Audio effect

  20. SchemeHandlers and Byte-StreamHandlers SchemeHandler Byte-StreamHandler ByteStream Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Audio Audio encoder Audio sink Audio decoder Audio effect

  21. What is a ByteStream? • Represents a byte stream from some data source: • Local file • Network file • Some other source • This interface supports typical stream operations such as reading, writing, and seeking.

  22. IMFByteStream Interface

  23. SchemeHandlers and Byte-StreamHandlers SchemeHandler Byte-StreamHandler ByteStream Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Audio Audio encoder Audio sink Audio decoder Audio effect

  24. SchemeHandlers and Byte-StreamHandlers SchemeHandler SourceResolver Byte-StreamHandler ByteStream Video Media source Video decoder Video effect Video encoder Video sink Media Foundation pipeline Audio Audio encoder Audio sink Audio decoder Audio effect

  25. Media Foundation pipeline SchemeHandler SourceResolver ByteStream Byte-StreamHandler Video Media source Video decoder Video effect Video encoder Video sink Audio Audio encoder Audio sink Audio decoder Audio effect

  26. Media Foundation pipeline SchemeHandler SourceResolver ByteStream Byte-StreamHandler Video Media source Video decoder Video effect Video encoder Video sink Audio Audio encoder Audio sink Audio decoder Audio effect

  27. Media Foundation pipeline SchemeHandler SourceResolver ByteStream Byte-StreamHandler Video Media source Video decoder Video effect Video encoder Video sink Audio Audio encoder Audio sink Audio decoder Audio effect

  28. How to build an extension using C++/CX The proper use of WRL to implement a SchemeHandler Instantiating a media source from a SchemeHandler Using the PropertySet class to implement two-way communication between the app and the extension PropertySet change notification to support dynamic updates Direct3D rendering Demo: Building a media extension • Marcin Stankiewicz • Development - Multimedia Platform Team

  29. Recap of Demo • How to build an extension using C++/CX • The proper use of WRL to implement a SchemeHandler • Instantiating a media source from a SchemeHandler • Using the PropertySet class to implement two-way communication between the app and the extension • PropertySet change notification to support dynamic updates • Direct3D rendering

  30. Media Foundation pipeline SchemeHandler SourceResolver ByteStream Byte-StreamHandler Video Media source Video decoder Video effect Video encoder Video sink Audio Audio encoder Audio sink Audio decoder Audio effect

  31. How to implement a sink media extension for audio Capturing uncompressed audio Processing audio buffers in a JavaScript app Writing audio to a wave file Asynchronous processing techniques Demo: Streaming media to an app • Matthew Howard • Development - Multimedia Platform Team

  32. Recap of Demo • How to implement a sink media extension for audio • Capturing uncompressed audio • Processing audio buffers in a JavaScript app • Writing audio to a wave file • Asynchronous processing techniques

  33. Need more information? Come to our question-and-answer session! 3-106 Chalk Talk: Media apps Thursday 4:15pm, B33 Hood

  34. Media-related sessions 3-117 10/31/2012 17:15 300 - Advanced B33 Hood Key technologies for building advanced media apps 3-122 11/1/2012 14:30 300 - Advanced B33 Hood Media app case studies 3-106 11/1/2012 16:15 300 - Advanced B33 Hood Chalk Talk: Media apps

  35. Windows 8 media overview http://blogs.msdn.com/b/b8/archive/2012/06/08/building-a-rich-and-extensible-media-platform.aspx Media extensions SDK sample http://code.msdn.microsoft.com/windowsapps/Media-extensions-sample-7b466096 Windows 8 multimedia desktop APIs available to Windows Store apps http://msdn.microsoft.com/en-us/library/windows/apps/hh452756.aspx SchemeHandlers and ByteStreamHandlers http://msdn.microsoft.com/en-us/library/windows/desktop/aa371872(v=vs.85).aspx Writing a Custom Media Source http://msdn.microsoft.com/en-us/library/windows/desktop/ms700134(v=vs.85).aspx Resources

  36. Resources • Develop: http://msdn.microsoft.com/en-US/windows/apps/br229512 • Design: http://design.windows.com/ • Samples: http://code.msdn.microsoft.com/windowsapps/Windows-8-Modern-Style-App-Samples • Videos: http://channel9.msdn.com/Windows Please submit session evals by using the Build Windows 8 app or at http://aka.ms/BuildSessions

More Related