Image Synthesis

1. Image Synthesis The programmable(D3D 10) Pipeline

2. User / Driver Vertex Stage Pixel Stage Texture 0 Texture 1 Texture 2 Texture 3 First: thefixedfunctionpipeline Vertex Stream Transform & Lighting Rasterizer Fragment Stream Texturing Blending/Ops

3. Fixed Function Pipeline

4. Fixed Function Pipeline • To In thefollowing: • Non-OpenGLeffectsusingfixedfunctionpipeline • Multipass • Combine pixelresultsof multiple renderingpasses • Pre-computedtexturemaps • Pre-computeresultsofcertain non-standardoperationsandstore in texturemaps

5. Advanced shading The Phong lighting model • N: surface normal • L: lightsourcedirection • R: reflectionvector • V: direction to the viewpoint • n:materials specular reflection exponent • Defines the highlights falloff from the direction of reflection • Specular reflections show the color of the light source • Not valid for all materials

6. Advanced shading The Phong/Blinn lighting model • The Halfway vector H points into the direction of a surfaces´ normal that reflects light into V • If light source and view point are considered to be at infinity, H is a constant • The angle between R and V is twice the angle between N and H if all above vectors are complanar • Highlights appear sharper in the Phong model

7. Advanced shading • In OpenGL, per-vertex Phong lighting is employed to simulate specular reflections • But, during scan-conversion vertex color is interpolated • OpenGL uses Gouraud shading instead of Phong shading • Texture color affects highlight color • Sharp highlights might be missed if the surface is tesselated coarsely

8. Advanced shading • To get separate highlight color highlights have to be added to the diffusely lit textured surface • OpenGL computes Ct(Cd + Cs) • Instead • Compute Cd and Cs separately • Modulate Cd with Ct • Add resultto Cs • Solution can be obtained by -blending

9. Advanced shading • Multipass technique to get separate highlight color • Render the textured object without specular lightCtCd • Render the untextured object with specular light only Cs • Combine the results in the frame buffer by additive blend • glBlendFunc(GL_ONE, GL_ONE) • glBlendEquation(GL_ADD)

10. Advanced shading Texturedwithdiffuse light Untextured with specular light Combined images In the original image the highlight can hardly be seen

11. Advanced shading • UsingOpenGL 1.2 glLightModel(GL_LIGHT_MODEL_COLOR_CONTROL,GL_SEPARATE_SPECULAR_COLOR);

12. A fewwords on OpenGLExtensions • The concept of extensions • Naming conventions • ARB vs. cross vendor vs. vendor specific extensions • Querying: • compile time vs. run time

13. OpenGL Extensions • Mechanism for providing access to non-standard features of the hardware • Every vendor may define own extensions without having to ask anybody else • Multiple vendors may cooperate to define more common extensions to ensure compatibility across platforms

14. Naming Conventions for Extensions • ARB: Architecture Review Board • Ext. that is likely to become part of core OpenGL in the future • EXT: Multi-vendor extensions • Ext. that multiple vendors have agreed upon • NV, ATI, SGI, SUN...: vendor-specific ext. • Ext. supported only by one vendor, sometimes experimental and not available on all systems of that vendor (e.g. NVX, SGIS, SGIX)

15. Dealing with Extensions • Since all extensions are optional, applications cannot rely on their presence • Have to check both at compile time and at run time! • Compile time check: #ifdefGL_ARB_multitexture // use multitexture ext. in here ... #endif

16. Dealing with Extensions (2) Run time check: const Glubyte *extstring = glGetString(GL_EXTENSIONS); strcpy(exts, extstring); extsupported= 0; next= strtok(exts, “ “); while(next) { if(!strcmp(next, “GL_ARB_multitexture”)) extsupported= 1; break; next= strtok(exts,NULL); }

17. Advanced shading How to get sharp highlights without adaptive refinement (tesselation) • Using the Blinn/Phong model specular highlights depend on L,N and V • By fixing L and V highlights depend only on N • N can be used to index into a texturemap in which the reflection for R = f(N,L) are stored • Texturemap looks like reflection from chrome hemisphere

18. Advanced shading Spherical texture maps Texel contain pre-computed Phong lighting Texture coordinates map into sphere map texture Texture coordinates computed from the normal vector

19. Advanced shading Phong shading with highlight texture • Render the object with diffuse light CtCd • Render the unlit, untextured white object and modulate it with the highlight texture Cht • Combine the results in the frame buffer by additive blend • … oruse multi-textures in onerendering pass

20. Advanced shading Phong shading with highlight texture Blended images Textured with diffuse light White object modulated by highlight texture

21. No more “Hacks” … The Programmable pipeline • Direct3D 9 / OpenGL 2.0 Model • Still used in many existing games • Supported by any graphics card of the last 5 years • Supported by every mayor OS(DX only on Windows) • Direct3D 10 / OpenGL 2.0 + Ext / OpenGL 3.0 • Supported only by the latest GPUsNVIDIA 8xxx or ATI Radeon 2xxxx • OpenGL 3.0 not out yet  must be used via extensions • Direct3D 10.0 only supported by Windows Vista(Direct3D 10.1 requires Vista SP1)

22. The Direct3D 9.0c Pipeline Input Data Memory Input Assembler Buffer Resources: Stage (IA) Buffers, Textures Vertex Shader Texture , Constants Stage (VS) Rasterizer Stage (RS) Texture, Constants Pixel Shader Stage (PS) States Output Merger Stage (OM) Output Data Texture

23. Features of the D3D 9.0c pipeline • Programmable vertex and pixel shaderwith limited shader program length (and many other limitations) • Vertex and pixel shader texture fetch capabilities (some limitations apply) • Configurable output merger (Blending/Ops/Tests) • Fixed function pipeline vertex and/or pixel shader can be used • Floating point through the entire pipeline (except for the blending)

24. Benefits of the programmable Pipeline • No more “hacks”  more sophisticated “hacks” • Straight forward program flow without the necessity of cryptic configuration changes • Decent knowledge of the transformation and lighting required to program the pipeline everything has to be done “by hand” • Shaders can be programmed in a high level language HLSL/GLSL

25. Shaders: a first look (HLSL) struct PS_OUTPUT { float4 RGBColor : COLOR0; // Pixel color }; PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform boolbTexture ) { PS_OUTPUT Output; // Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = tex2D(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse; return Output; } technique RenderSceneWithTexture1Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) } } technique RenderSceneWithTexture2Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 2, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) } } technique RenderSceneWithTexture3Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 3, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) } } technique RenderSceneNoTexture { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, false, false ); PixelShader = compile ps_2_0 RenderScenePS( false ); // trivial pixel shader (could use FF instead if desired) } } “Basic HLSL”-Sample from the DirectX SDK: //-------------------------------------------------------------------------------------- // Global variables //-------------------------------------------------------------------------------------- float4 g_MaterialAmbientColor; // Material's ambient color float4 g_MaterialDiffuseColor; // Material's diffuse color intg_nNumLights; float3 g_LightDir[3]; // Light's direction in world space float4 g_LightDiffuse[3]; // Light's diffuse color float4 g_LightAmbient; // Light's ambient color texture g_MeshTexture; // Color texture for mesh float g_fTime; // App's time in seconds float4x4 g_mWorld; // World matrix for object float4x4 g_mWorldViewProjection; // World * View * Projection matrix //-------------------------------------------------------------------------------------- // Texture samplers //-------------------------------------------------------------------------------------- sampler MeshTextureSampler = sampler_state { Texture = <g_MeshTexture>; MipFilter = LINEAR; MinFilter = LINEAR; MagFilter = LINEAR; }; //-------------------------------------------------------------------------------------- // Vertex shader output structure //-------------------------------------------------------------------------------------- struct VS_OUTPUT { float4 Position : POSITION; // vertex position float4 Diffuse : COLOR0; // vertex diffuse color (note that COLOR0 is clamped from 0..1) float2 TextureUV : TEXCOORD0; // vertex texture coords }; //-------------------------------------------------------------------------------------- // This shader computes standard transform and lighting //-------------------------------------------------------------------------------------- VS_OUTPUT RenderSceneVS( float4 vPos : POSITION, float3 vNormal : NORMAL, float2 vTexCoord0 : TEXCOORD0, uniform intnNumLights, uniform boolbTexture, uniform boolbAnimate ) { VS_OUTPUT Output; float3 vNormalWorldSpace; float4 vAnimatedPos = vPos; // Animation the vertex based on time and the vertex's object space position if( bAnimate ) vAnimatedPos += float4(vNormal, 0) * (sin(g_fTime+5.5)+0.5)*5; // Transform the position from object space to homogeneous projection space Output.Position= mul(vAnimatedPos, g_mWorldViewProjection); // Transform the normal from object space to world space vNormalWorldSpace = normalize(mul(vNormal, (float3x3)g_mWorld)); // normal (world space) // Compute simple directional lighting equation float3 vTotalLightDiffuse = float3(0,0,0); for(inti=0; i<nNumLights; i++ ) vTotalLightDiffuse += g_LightDiffuse[i] * max(0,dot(vNormalWorldSpace, g_LightDir[i])); Output.Diffuse.rgb = g_MaterialDiffuseColor * vTotalLightDiffuse + g_MaterialAmbientColor * g_LightAmbient; Output.Diffuse.a = 1.0f; // Just copy the texture coordinate through if( bTexture ) Output.TextureUV = vTexCoord0; else Output.TextureUV = 0; return Output; }

26. Vertex Shaders: a closer look VS_OUTPUT RenderSceneVS( float4 vPos : POSITION, float3 vNormal : NORMAL, float2 vTexCoord0 : TEXCOORD0, uniform intnNumLights, uniform boolbTexture, uniform boolbAnimate ) { VS_OUTPUT Output; float3 vNormalWorldSpace; float4 vAnimatedPos = vPos; // Animation the vertex based on time and the vertex's object space position if( bAnimate) vAnimatedPos += float4(vNormal, 0) * (sin(g_fTime+5.5)+0.5)*5; // Transform the position from object space to homogeneous projection space Output.Position = mul(vAnimatedPos, g_mWorldViewProjection); // Transform the normal from object space to world space vNormalWorldSpace = normalize(mul(vNormal, (float3x3)g_mWorld)); // normal (world space) // Compute simple directional lighting equation float3 vTotalLightDiffuse = float3(0,0,0); for(inti=0; i<nNumLights; i++ ) vTotalLightDiffuse += g_LightDiffuse[i] * max(0,dot(vNormalWorldSpace, g_LightDir[i])); Output.Diffuse.rgb = g_MaterialDiffuseColor * vTotalLightDiffuse + g_MaterialAmbientColor* g_LightAmbient; Output.Diffuse.a = 1.0f; // Just copy the texture coordinate through if( bTexture ) Output.TextureUV = vTexCoord0; else Output.TextureUV = 0; return Output; }

27. Pixel Shaders: a closer look PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform boolbTexture ) { PS_OUTPUT Output; // Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = tex2D(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse; return Output; }

28. Putting it all together: Effect Files • Store many vertex and pixel shaders as well as pipeline stages in a single file • Also Store variables, textures, and samplers • Allow for grouping of shaders into techniques and passes • Can be accessed easily from the main program • Can be created/modified easily from a content creation tool(AMD RenderMonkey/Nvidia FX Composer)

29. FX Composer Showcase

30. Effects in the “Basic HLSL”-Sample //-------------------------------------------------------------------------------------- // Global variables //-------------------------------------------------------------------------------------- float4 g_MaterialAmbientColor; // Material's ambient color float4 g_MaterialDiffuseColor; // Material's diffuse color intg_nNumLights; float3 g_LightDir[3]; // Light's direction in world space float4 g_LightDiffuse[3]; // Light's diffuse color float4 g_LightAmbient; // Light's ambient color float g_fTime; // App's time in seconds float4x4 g_mWorld; // World matrix for object float4x4 g_mWorldViewProjection; // World * View * Projection matrix

31. Effects in the “Basic HLSL”-Sample texture g_MeshTexture; // Color texture for mesh //-------------------------------------------------------------------------------------- // Texture samplers //-------------------------------------------------------------------------------------- sampler MeshTextureSampler = sampler_state { Texture = <g_MeshTexture>; MipFilter = LINEAR; MinFilter = LINEAR; MagFilter = LINEAR; };

32. Effects in the “Basic HLSL”-Sample //-------------------------------------------------------------------------------------- // Renders scene to render target //-------------------------------------------------------------------------------------- technique RenderSceneWithTexture1Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 1, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) } }

33. Effects in the “Basic HLSL”-Sample technique RenderSceneWithTexture2Light { pass P0 { VertexShader = compile vs_2_0 RenderSceneVS( 2, true, true ); PixelShader = compile ps_2_0 RenderScenePS( true ); // trivial pixel shader (could use FF instead if desired) } }

34. The DX 10.0 Pipeline Input Data Memory Input Assembler Buffer Resources: Stage (IA) Buffers, Textures, Constant Buffers Vertex Shader Texture, Constant Buffer Stage (VS) Geometry Shader Texture, Constant Buffer Stage (GS) Stream Output Buffer Stage (SO) Rasterizer Stage (RS) Texture, Constant Buffer Pixel Shader Stage (PS) States Output Merger Stage (OM) Output Data Buffer, Texture, Constant Buffer

35. Features of the D3D 10.1 pipeline • Programmable vertex, pixel, and geometry shaderwith practically unlimited shader program length • Vertex, pixel, and geometry shader texture fetch capabilities • Configurable output merger (Blending/Ops/Tests) • Fixed function pipeline vertex and/or pixel shader can NOT be used anymore • Floating point through the entire pipeline including blending • API Support for double precision (first hardware supporting double expected to ship in late 2007)

36. “Basic HLSL” D3D 10 /-------------------------------------------------------------------------------------- // This shader outputs the pixel's color by modulating the texture's // color with diffuse material color //-------------------------------------------------------------------------------------- PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform boolbTexture ) { PS_OUTPUT Output; // Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = g_MeshTexture.Sample(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse; return Output; } //-------------------------------------------------------------------------------------- // Renders scene to render target using D3D10 Techniques //-------------------------------------------------------------------------------------- technique10 RenderSceneWithTexture1Light { pass P0 { SetVertexShader( CompileShader( vs_4_0, RenderSceneVS( 1, true, true ) ) ); SetGeometryShader( NULL ); SetPixelShader( CompileShader( ps_4_0, RenderScenePS( true ) ) ); SetDepthStencilState( EnableDepth, 0 ); } }

37. “Basic HLSL” D3D 10 /-------------------------------------------------------------------------------------- // This shader outputs the pixel's color by modulating the texture's // color with diffuse material color //-------------------------------------------------------------------------------------- PS_OUTPUT RenderScenePS( VS_OUTPUT In, uniform boolbTexture ) { PS_OUTPUT Output; // Lookup mesh texture and modulate it with diffuse if( bTexture ) Output.RGBColor = g_MeshTexture.Sample(MeshTextureSampler, In.TextureUV) * In.Diffuse; else Output.RGBColor = In.Diffuse; return Output; } //-------------------------------------------------------------------------------------- // Renders scene to render target using D3D10 Techniques //-------------------------------------------------------------------------------------- technique10 RenderSceneWithTexture1Light { pass P0 { SetVertexShader( CompileShader( vs_4_0, RenderSceneVS( 1, true, true ) ) ); SetGeometryShader( NULL ); SetPixelShader( CompileShader( ps_4_0, RenderScenePS( true ) ) ); SetDepthStencilState( EnableDepth, 0 ); } }

38. Coming up GPU Effects