Graphics at the core of Windows 8 and your app

1. Graphics at the core ofWindows 8 and your app Shai Hinitz Sr. Program Manager, Windows Games 3-112

2. Session agenda • Windows 8 graphics overview • Graphics development language options • Migrating your graphics to WinRT • Case Study: Cat in the Hat by Oceanhouse Media

3. Windows 8 is fully graphics hardware accelerated to increase performance and reduce power consumption • The Lock and Start screen • Platform for “fast and fluid” & “stick to your finger” experiences • Internet Explorer 10 • Rich visual and multimedia experiences • All Windows Store apps are hardware accelerated • All rendering surfaces are built on DirectX • Use DirectX for maximum control of GPU performance and extensibility

4. 3D graphics Segoe UI 2D graphics Image & video decoding Fonts, text analysis, layout GPGPU Too

5. Direct3D Segoe UI HTML5, CSS3 & Direct2D HTML5, Direct2D effects, WIC & Media Foundation HTML5, CSS3 & DirectWrite DirectCompute & C++AMP

6. Graphics language of choice • Write full-screen, chrome-free applications with smooth, flicker-free action using HTML5 and CSS3 Windows Store apps UI toolkit • Develop interactive 2-D applications and include 3-D graphics using XAML Windows Store apps UI toolkit • Author fully immersive interactive 3-D experiences by taking advantage of the full processing capabilities of modern graphics hardware using HLSL with DirectX® 11.1 • Use hybrid models supporting XAML & DirectX if appropriate

7. Development language combinations • HTML/CSS for GUI/Markup and canvas for rendering • Leverage hardware accelerated engine from IE • Windows features (file I/O, events, notifications, etc.) are exposed in JavaScript • XAML can be used with C++ or C# or Visual Basic apps • DirectX can be used with C++ (or C# via third-party wrappers) • Hybrid models supported • DirectX and XAML • JavaScript wrappers for DLLs

8. HTML5/JavaScript code snippet • Javascript • // get the canvas and the 2D context for it • varcanvasElement = document.getElementById("myCanvas"); • var context = canvasElement.getContext("2d"); • // set the font and size • context.font = "20px Georgia"; • // clear an area and render hello world at position (10,50) • context.clearRect(0.0.300,150); • context.fillText("Hello World!", 10, 50); • HTML • // define the canvas size with the tag in the HTML doc • <canvasid="myCanvas" width ="720" height ="486"></canvas>

9. XAML/C# code snippet • XAML • // Set rendering location and text style • <StackPanelGrid.Row="1" Margin="120,30,0,0"> • <TextBlock x:Name="greetingOutput" Style="{StaticResourceShyTextStyle}"/> • </StackPanel> • C# • // Set text • privatevoidYourFunction(object sender, Windows.UI.Xaml.RoutedEventArgs e) • { • greetingOutput.Text = "Hello World!"; • }

10. DirectX/C++ code snippet • voidDWriteHelloWorld::CreateWindowSizeDependentResources() • { • // Create the text string • DirectXBase::CreateWindowSizeDependentResources(); • Platform::String^ text = "Hello World!"; • D2D1_SIZE_F size = m_d2dContext->GetSize(); • // Create a DirectWrite Text Layout object • DX::ThrowIfFailed( • m_dwriteFactory->CreateTextLayout( • text->Data(), // Text to be displayed • text->Length(), // Length of the text • m_textFormat.Get(), // DirectWrite Text Format object • size.width, // Width of the Text Layout • size.height, // Height of the Text Layout • &m_textLayout • ) • );

11. DirectX/C++ code snippet • voidDWriteHelloWorld::Render() • {  • // Clear the buffer and set up • m_d2dContext->BeginDraw(); • m_d2dContext->Clear(D2D1::ColorF(D2D1::ColorF::CornflowerBlue)); • m_d2dContext->SetTransform(D2D1::Matrix3x2F::Identity()); • // Update text position and style • m_d2dContext->DrawTextLayout( •        D2D1::Point2F(0.0f, 0.0f), • m_textLayout.Get(), • m_blackBrush.Get() • ); • m_d2dContext->EndDraw(); • }

12. Desktop apps Windows Store apps View XAML HTML/CSS HTML JavaScript C# VB C C++ JavaScript C# VB Model controller C C++ Where’s DirectX?? WinRT APIs System services Devices and printing Communication and data DirectX graphics and media Application model .NET Silverlight Internet Explorer Win32 Windows kernel services Kernel

13. Graphics architecture in Windows 8 HTML, CSS, and XAML graphics C++ AMP Direct2D Media Foundation DirectCompute Direct3D Video DXGI Graphics Processing Unit (GPU)

14. Windows 8 DirectX advantages • The new Windows 8 graphics stack is better integrated • Unified graphics runtime avoids side by side installation issues • Feature levels target the broadest range of GPU levels and vendors • For the ultimate experience in graphics and video, use DirectX 11.1 features to bring tessellation and stereoscopic 3D to your apps

15. 3D graphics via Direct3D 11 • Use the same API for Windows Store and desktop apps • WinRT API supports the newest DirectX 11.1 features • Support for multiple hardware generations via Feature Levels • Feature_Level_9 • Feature_Level_10 (plus all of 9) • Feature_Level_11 (plus all of 10) • Direct3D 11.1 has new updates for Windows 8 • Performance improvements • Higher quality visuals • Power efficiency • Ease of development Check out 3-113 Graphics with the Direct3D11.1 API made easy

16. Porting your apps to the Windows Runtime

17. Porting graphics apps to WinRT • Determine source  target technology • Web  HTML • Silverlight/WPF  XAML • OpenGL  Direct3D • Reuse art assets (images, textures, fonts) • Get the uncompressed (un-flattened) sources (e.g. PSD) • Source vector-based images • Preserve graphics code and mark up resources • HTML, CSS, JavaScript, XAML, C#, VB, C++… • Start from your app or a template or sample?

18. General porting approach • Adapt app’s UI to exploit Windows 8 features and follow design guidelines • Migrate your code from JavaScript/C#/Visual Basic/C++ to the corresponding Windows Runtime APIs • Start with VS template that best matches your app layout and functionality Copy folders, code and art assets to reuse to new project Include the files in the new Visual Studio Project Copy the reusable markup code (HTML, CSS, XAML, shaders) code into the new project

19. General porting approach • Search and replace source namespaces with target WinRT namespaces • Build the project to generate errors to locate code to update or replace • Stub out any code that you cannot easily update until your project builds • Replace the commented-out code line by line until your port is complete

20. Web  WinRT JavaScript/HTML

21. Key considerations • WWAHost.exe hosts and executes your app • JavaScript content should just work • Use JavaScript to call WinRT APIs • iFrames supported – note local vs. hosted content • Some web technologies are not supported • ActiveX controls • Java objects • See Migrating a web app using JavaScript and HTML on MSDN

22. Graphics considerations • <canvas> for real-time graphics controlled by JavaScript • Support for HTML5, CSS3 and SVG content for general graphics and UI • Most video, audio media and image content/tags will be compatible • 2D and 3D transforms for 2D and 2½D graphics • Concatenation of multiple transforms for improved performance • Some web graphics technologies are not supported • Non HTML5 standard objects • Adobe Flash Player • Silverlight content transitions best to XAML Check out 3-110 From zero to hero! Building a Windows Store game in HTML5

23. Silverlight/WPF  WinRT C#/XAML

24. Key considerations • Leverage your experience from Silverlight and WPF • Programming concepts and usage patterns influenced WinRT design • Porting effort will depend on breadth of APIs used • WinRT APIs include most commonly used subset of .NET framework APIs • Much XAML and C# or Visual Basic will transition smoothly • Search and replace "System.Windows“ with (WinRT)"Windows.UI.Xaml"

25. Graphics considerations • Code runs directly on the runtime (no framework host) • Input handled directly • More secure than internet-centric model • Less storage limitations • WebView can host HTML/CSS content • Asynchronous approach for best performance • The WinRT imaging API supports more image source formats • Some methods that would slow down rendering (like OpacityMask) were omitted • Easy to use encoder/decoder types • New theme and transition animations • Similar media APIs with DRM plug-in options Check out 3-116 Introduction to creating Windows Store apps using XAML

26. OpenGL|ES WinRT C++ /

27. OpenGL and Direct3D similarities • Both have similar rendering pipelines and graphics features • Direct3D is a rendering implementation and API, not a spec • Hardware accelerated at all times • OpenGL is a rendering spec and API, not an implementation • Hardware accelerated where possible

28. General differences • DirectX • Graphics API and runtime library for Windows Store apps • Lower modules map directly to GPU components • Direct access to hardware layout; app can precisely manage resources and processing • Higher-level modules (Direct2D, MPT) built upon lower modules to simplify development • IHV consistency via certification • OpenGL • Rendering spec and multi-vendor implementation for Desktop apps • Hardware-agnostic rendering pipeline abstraction • Abstracted for hardware diversity, runtime manages most resources • OpenGL provides higher-level modules via 3rd party libraries (e.g. SDL) • IHV differentiation via extensions

29. Shader comparisons • GLSL (OpenGL) • Procedural, step-centric (C like) • Remain legible for compilation at run time • Row-major matrices (default) • Typical vector type: vec2/3/4 • texture2D [function] • sampler2D [datatype] • lowp, mediump, highp • uniform variables • Varying passes data between pipeline stages HLSL (Direct3D) Object Oriented, data-centric (C++ like) Precompiled for performance and obfuscation Column-major matrices (default) Typical vector type: float2/3/4 texture.Sample [method] Texture2D [datatype] min10float, min16float constant buffers Data transfers via input layout declarations

30. Optional DirectX components • High-speed processing with DirectXMath: • C++ SIMD graphics math library (calculating T, S, R transformation matrices) • Includes SSE2 and ARM-NEON extensions • Common texture operations with DirectXTex: • Leverages Windows Image Component (WIC) APIs • Resizing, format conversion, mipmap generation • Read/write DDS files • Encode/decode Block compression (BC) textures for Direct3D runtime • Height-map to normal-map conversion • Simple .TGA reader/writer

31. Optional DirectX Took Kit (DirectXTK) • SpriteBatch - simple and efficient 2-D sprite rendering • SpriteFont – bitmap-based text rendering • Effects – set of built-in shaders for common rendering tasks • GeometricPrimitive – draws basic shapes such as cubes and spheres • CommonStates – factory providing commonly used D3D state objects • VertexTypes – structures for commonly used vertex data formats • DDSTextureLoader – lightweight DDS file texture loader • WICTextureLoader – WIC-based image file texture loader • ScreenGrab – lightweight screen shot saver

32. OpenGL/DirectX code comparison

33. Vertex shader OpenGL example • //The incoming vertex' position • attribute vec4 position; • //And its color • attribute vec3 color; • //The varying statement tells the shader pipeline that this variable • //has to be passed on to the next stage (to the fragment shader) • varying vec3 colorVarying; • //The shader entry point is the main method • void main() • { • colorVarying = color; //Pass the color to the fragment shader • gl_Position = position; //Copy the position • }

34. Vertex shader Direct3D example • // Define the vertex and pixel shader structs • structVertexShaderInput • { •     float3 pos : POSITION; •     float4 color : COLOR; • }; • structPixelShaderInput • { •     float4 pos : SV_POSITION; •     float4 color : COLOR; • }; • // Pass position and color values from Vertstruct to Pixel struct (adding W and Alpha) • PixelShaderInputSimpleVertexShader(VertexShaderInput input) • { • PixelShaderInputvertexShaderOutput; • vertexShaderOutput.pos = float4(input.pos, 1.0f); • vertexShaderOutput.color = float4(input.color, 1.0f);  • returnvertexShaderOutput; • }

35. Fragment shader OpenGL example • varying vec3 colorVarying; • void main() • { • //Create a vec4 from the vec3 by padding a 1.0 for alpha • //and assign that color to be this fragment's color • gl_FragColor = vec4(colorVarying, 1.0); • }

36. Pixel shader Direct3D example • // Collect input from vertex shader • structPixelShaderInput • { •     float4 pos : SV_POSITION; •     float4 color : COLOR; • }; • // Set the pixel color value for the specified Renter Target • float4 SimplePixelShader(PixelShaderInput input) : SV_TARGET • { • returninput.color; • }

37. Rendering OpenGL example • // Bind shaders to pipeline (both VS and FS are in a program) • glUseProgram(m_shader->getProgram()); • // (Input Assembly) Get attachment point for position and color attributes • m_positionLocation = glGetAttribLocation(m_shader->getProgram(), "position“); • glEnableVertexAttribArray(m_positionLocation); • m_colorLocation = glGetAttribColor(m_shader->getProgram(), “color“); • glEnableVertexAttribArray(m_colorLocation); • // Bind the vertex buffer object • glBindBuffer(GL_ARRAY_BUFFER, m_geometryBuffer); • glVertexAttribPointer(m_positionLocation, 4, GL_FLOAT, GL_FALSE, 0, NULL); • glBindBuffer(GL_ARRAY_BUFFER, m_colorBuffer); • glVertexAttribPointer(m_colorLocation, 3, GL_FLOAT, GL_FALSE, 0, NULL); • // Draw a triangle of 3 vertices! • glDrawArray(GL_TRIANGLES, 0, 3);

38. Rendering Direct3D example • // Binding Shaders to pipeline (VS and PS) • m_d3dDeviceContext->VSSetShader(vertexShader.Get(),nullptr,0); • m_d3dDeviceContext->PSSetShader(pixelShader.Get(),nullptr,0); • // Declaring the expected inputs to the shaders • m_d3dDeviceContext->IASetInputLayout(inputLayout.Get()); • m_d3dDeviceContext->IASetVertexBuffers(0, 1, vertexBuffer.GetAddressOf(), &stride, &offset); • // Set primitive’s topology • m_d3dDeviceContext->IASetPrimitiveTopology(D3D11_PRIMITIVE_TOPOLOGY_TRIANGLELIST); • // Draw a triangle of 3 vertices! • m_d3dDeviceContext->Draw(ARRAYSIZE(triangleVertices),0);

39. Cat In The HatPorting Oceanhouse Media’s book engine to WinRT Ken Maffei Lead Programmer

40. Oceanhouse Media book titles • Dr. Seuss Collection • Little Critter Collection • Smithsonian • 5 Little Monkeys • Berenstain Bears

41. Path to WinRT • Android (Java), 2D Canvas • WebOS (C++), OpenGL • Ported the WebOS code to WinRT • App is matrix-driven for bitmaps and text

42. Ramp-up to WinRT • Direct port elements • Obviously had to change • Potential re-write using new WinRT concepts • Three categories • of effort

43. First things first – item #3 • Identify where the answers were • Confidence we could implement everything when needed • Confidence in rewrite items

44. WinRT sample projects • The samples provided a great resource: • Playing back multimedia • Reading up font collections from disk • Adding effects, such as outlining, to text • Often, cut and paste just works! • The samples • are invaluable

45. API documentation • Documentation includes sample code and descriptions of concepts such as: • The Asynchronous model • File I/O • Making HTTP requests • Familiarize yourself • with the API documentation

46. Some WinRT coding is low-level • WinRT API is powerful and flexible but… • Certain tasks may require a fair amount of code • Good News! The sample code always worked and easily modified • Sometimes closer to the metal than you are used to

47. Beginning the port • Deferring some features allowed smooth C++ code transition • Type conversions (Rects, Points and Bitmaps) were straightforward • Switched from UTF8 and std::string to Unicode and Platform::String^ • There were also some very unfamiliar types… • C++ code comes straight across, but there were lots of type conversions

48. Local storage and asynchronous file I/O • // Get the user application data folder • Windows::Storage::StorageFolder^ localFolder = ApplicationData::Current->LocalFolder; • // Create an asynch task to open the file • concurrency::task<StorageFile^> getFileOperation(localFolder->GetFileAsync(fileName)); • // Read the contents of the file asynchronously • getFileOperation.then([this, m_d2dContext](StorageFile^ file){ • return FileIO::ReadBufferAsync(file); • }).then([this, m_d2dContext](concurrency::task<IBuffer^> previousOperation) { • ... • }

49. Some things were just easier in WinRT! • Memory: no code forking required • Large bitmaps are not a problem • Animated Text: high-quality direct manipulation with Direct2D and DirectWrite • Direct2D • simplified many tasks for us

50. Text rendering was finally easy • // In Direct2D/DirectWrite, manipulating text is • // equivalent to manipulating bitmaps • // Set up scale, rotation & translation transforms • // These can be animated each frame • D2D1::Matrix3x2F r = D2D1::Matrix3x2F::Rotation(angle);D2D1::Matrix3x2F s = D2D1::Matrix3x2F::Scale(scale.x, scale.y);D2D1::Matrix3X2F t = D2D1::Matrix3X2F::Translate(x, y);m_d2dContext.SetTransform(s*r*t); • // If bitmap • m_d2dContext->DrawBitmap(bitmap.Get(), args); • // If text • m_textLayout.Get()->Draw(m_d2dContext, args);