Combining XAML and DirectX

1. Combining XAML and DirectX Jesse Bishop Program Manager, Windows XAML team Microsoft Corporation

2. Review: XAML in Windows 8 Your Metro-style UI framework Built in and custom controls Layout, styling, databinding Media, animation, drawing Design-time tools A focus on performance Leveraging hardware Independent animations Fast rendering and composition

3. Review: DirectX in Windows 8 Your graphics solution Everything in Windows 8 is built around DirectX: hardware to OS to UI frameworks Control vs. Complexity A tradeoff between difficulty and power of lower level APIs More work to align with Metro style guidelines

4. Rich UI + Real-time Graphics

5. The Best of Both Worlds XAML DirectX Interactivity, control model Highest performance rendering UI, drawing, animations Direct device access Data binding, media, HTML Complex 2D/3D graphics, text Design-time experience Image effects

6. When do you want UI + Graphics? Example scenarios: Visual design, modeling, creativity Complex image processing Large documents, maps Simulations Games 1. DirectX graphics added to a primarily XAML UI 2. Large-scale DirectX graphics mixed with some UI 3. DirectX-focused app with minimal UI • A one-size-fits-all approach wasn’t sufficient across the broad range of scenarios. • So, we added 3 XAML types which each target a different need.

7. Interop Functionality Integration Easy composition of XAML + DirectX using familiar XAML model All functionality of both XAML and DirectX Performance Low-latency input and interactivity Incremental rendering for both XAML and DirectX content Low overhead: retained-mode XAML and immediate-mode DirectX together C++ Only XAML code-behind can be managed, but interop DirectX is C++ only (without wrappers) Query for native interfaces of XAML WinRT types to access interop methods

8. 1. DirectX graphics added to a primarily XAML UI 2. Large-scale DirectX graphics mixed with some UI 3. DirectX-focused app with minimal UI

9. SurfaceImageSource A XAML ImageSource type. You can draw any 2D/3D content with DirectX then use it to fill any XAML element. Framework supplies a DX surface you draw to whenever you want to update content. Conceptually similar to WPF’s D3DImage – but easier and more powerful Fully integrated with XAML composition: z-index, transforms, animation. No “airspace” issues <Ellipse x:Name="Ellipse1" Stroke="Black" Height="300" Width="300" /> ImageBrush^ brush = ref new ImageBrush(); brush->ImageSource = ref new SurfaceImageSource(300, 300); Ellipse1->Fill = brush;

10. SurfaceImageSource Definition interfaceISurfaceImageSourceNative: IUnknown { SetDevice(IDXGIDevice*pDevice); BeginDraw( RECT updateRect, IDXGISurface** pSurface, POINT* offset ); EndDraw(); };

11. demo SurfaceImageSource

12. 1. DirectX graphics added to a primarily XAML UI 2. Large-scale DirectX graphics mixed with some UI 3. DirectX-focused app with minimal UI

13. VirtualSurfaceImageSource Extends SurfaceImageSource concept to support huge virtualized areas. Useful when content is larger than what fits on the screen. Potentially MUCH larger. Uses a tiling + callback approach. Content is split into rectangular regions. Framework requests updates from your app when it needs to render a region. Performance characteristics are tuned for large content. Automatically integrates with XAML ScrollViewer so it knows what’s currently visible. Manages a smart cache of tiles.

15. VirtualSurfaceImageSource Definition interface IVirtualSurfaceImageSourceNative: ISurfaceImageSourceNative { RegisterForUpdatesNeeded(IVirtualSurfaceUpdatesCallbackNative *pCallback); GetUpdateRects(RECT*pUpdates, DWORD count); GetUpdateRectCount(DWORD*pCount); Invalidate(RECTupdateRect); /* ... */ }; interfaceIVirtualSurfaceUpdatesCallbackNative: IUnknown { UpdatesNeeded(); };

16. VirtualSurfaceImageSource Usage 1. Register your UpdatesNeeded() callback This gets called when a region of the surface needs to be updated Get the list of tiles that need to be updated with GetUpdateRects() Draw updates using the same BeginDraw()/EndDraw() approach If a region needs an update, Invalidate() it Especially for off-screen content, since you can also update using BeginDraw()/EndDraw() You can dynamically resize the total area as needed 2. 3. 4. 5.

17. 1. DirectX graphics added to a primarily XAML UI 2. Large-scale DirectX graphics mixed with some UI 3. DirectX-focused app with minimal UI

18. SwapChainBackgroundPanel Full-screen DirectX app with XAML content as an overlay. Inherits from Grid control to enable easy XAML layout Full control of DirectX surface. Enables low latency interaction and control over present timing Not required to be tied to the XAML UI thread, though it can be Tradeoff between more control vs. closer integration with XAML scene. Full-screen only Must be the root element of an app XAML child elements are drawn on top

19. SwapChainBackgroundPanel Definition interface ISwapChainBackgroundPanelNative: IUnknown { SetSwapChain(IDXGISwapChain *pSwapChain); };

20. Using SwapChainBackgroundPanel <SwapChainBackgroundPanelx:Class="MyApp.MainPage" x:Name="SwapChainPanel" PointerMoved="OnPointerMoved"> <TextBlockText="Here's some XAML content!" /> </SwapChainBackgroundPanel>

21. Creating the Swap Chain CreateSwapChainForComposition dxgiFactory-> ( m_d3dDevice.Get(), &swapChainDesc, nullptr, &m_swapChain ); ComPtr<> panelNative; reinterpret_cast<IUnknown*>(SwapChainPanel)->QueryInterface( IID_PPV_ARGS(&panelNative) ); panelNative-> (m_swapChain.Get()); ISwapChainBackgroundPanelNative SetSwapChain

22. SwapChainBackgroundPanel Tips Adding XAML children Be mindful of overdraw: scope XAML element sizes to just the area required to display content Use DebugSettings::IsOverdrawHeatMapEnabled to visualize overdraw Drawing content Create a landscape swap chain and apply rotation as needed CompositionTarget::Rendering event for real-time updates on XAML UI thread Async queue to a 2nd DX device for long computation or render tasks Input handling Can use normal XAML input event handlers on the SwapChainBackgroundPanel element Xinput for controllers if applicable

23. App Composition • DX • XAML SwapChainBackgroundPanel SurfaceImageSource

24. demo SwapChainBackgroundPanel

25. Recap Three approaches to combining XAML + DirectX SurfaceImageSource, VirtualSurfaceImageSource, SwapChainBackgroundPanel You can mix and match in the same app Consider the scenario you’re targeting XAML alone will work great for many apps – retained-mode graphics with smooth 60 fps animations SurfaceImageSource for close integration with XAML composition at a per-element level VirtualSurfaceImageSource for scrolling/zooming large-scale DirectX content SwapChainBackgroundPanel for adding UI to full-screen DirectX games and apps Structuring your app Define your Metro-style UI using XAML markup Use XAML code-behind for UI-related processing such as input handling and view state changes Wrapping interop code in a native WinRT DLL allows it to be used from managed code as well Query XAML type for its native interface to access DirectX interop methods

26. Resources Blog: Combining XAML and DirectX http://blogs.msdn.com/b/windowsappdev/archive/2012/03/15/combining-xaml-and-directx.aspx MSDN: DirectX and XAML interop http://msdn.microsoft.com/en-us/library/windows/apps/hh825871.aspx DirectX + XAML 3D game sample app http://code.msdn.microsoft.com/windowsapps/Metro-style-DirectX-18f98448 Other sample apps http://code.msdn.microsoft.com/windowsapps/Magazine-Sample-2a657289 http://code.msdn.microsoft.com/windowsapps/Direct2D-lighting-effects-e0801da3

27. © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.