95.4501

1. 95.4501 Advanced Detailing

2. References • Rendering Surface Details in Games with Relief Mapping Using a Minimally Invasive Approach, Policarpo and Oliveira, pp 109-119, ShaderX4, 2006. • Practical Parallax Occlusion Mapping with Approximate Soft Shadows for Detailed Surface Rendering, Tatarchuk, pp 75-105, ShaderX5, 2007. • Relief Mapping of Non-Height-Field Surface Details, Policarpo and Oliveira, Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games, pp. 55–62. • Relaxed Cone Stepping for Relief Mapping, Policarpo and Oliveira, GPU Gems 3, pp. 409–428.

3. 95.4501 Introduction

4. Advanced Detailing? a flat surface making it “bumpy”

5. Advanced Detailing? Advanced Detailing:Using shaders and special textures to make flat surfaces appear 3-dimensional… Two techniquesBump mapping: A pure side effect of lightingProblem: Silhouettes (grazing view) are flat. • Displacement mapping: A 3D effect independent of lighting; originally done by materializing actual geometry but more recently done via geometry shaders. PolicarpoBumpmapping Demo Not yet a topic in the course

6. Techniques Simple:Procedurally perturb normals via noise or via modifications to the color texture itself. Normal textures: Provide entire normals via a separate texture. Normal + height textures: Provide both normalsand heights via one or more separate textures. With geometyshaders, can build geometry from normal + height textures:Fast enough for demos but perhaps not fast enough for games (certainly does not work on IPhone, xBox, or Playstation)

7. Parallax Parallax: the difference in apparent position as you move from side to side : • What some people call “the 3D effect”; namely that near objects MOVE MORE than far objects. From wikipedia

8. Artistic Issues Is the effect barely noticeable?Is it just a perturbation or more sophisticated ray tracing pullout effect. Does it self-occlude?Is a bump occluded by another in front of it? Does it self-shadow?Is bump valley dark when the light is occluded by a bump mountain? Does it have a flat silhouette?Are grazing angles flat? Is it expensive?Does it make the game crawl?

9. Increasingly Realistic Bumpmapping Techniques Simple bump mapping (Blinn 78):Using generated normalsORnormals computed from the original color texture. Normal mapping (Cohen 78, Cignoni 78):Usingnormals stored in a texture. Can’t self-shadow Parallax mapping (Kaneko 01):Usingnormals + depth stored in a texture to simulate parallax in one-pass.

10. Increasingly Realistic Bumpmapping Techniques Relief Mapping (Policarpo05):Using textures with normals + depth + iterative ray tracing techniques to allow for better occlusion and more depth. Cone Mapping (Dummer 06):Using cones to indicate how far you can step before you need to check for possible intersection.

11. Implementation Issues Texture encodings? How do we store and access the normal and optional height information in a texture. Texture coordinate space? • How many spaces do shaders have to deal with? Texture coordinate space, local space (for a model), world space, camera space, perspective space… Ray tracing? Do shaders loop and if so, do they need to perform linear and binary searches.

12. 95.4501 Texture Encodings

13. Encoding Normal [nx, ny, nz] in 8-bit RGB Textures What is often done: Quantize each float value in range [-1.0, +1.0] by mapping to range [0.0, 255.0] and flooring to convert to integer. 0.0 1.0 254.0 255.0 floor floor Very few values map to 255 0 255 1 254

14. Encoding Normals in Textures Fairer technique: Quantize each float value in range [-1.0, +1.0] to range [0.0, 256.0], floor to convert to integer, then map 256.0 to 255. 0.0 1.0 255.0 256.0 floor floor As many float values from 255.0 to 256.0 map to 255as from 0.0 to 1.0 map to 0 0 255 1

15. Encoding Normals in Textures • Can encode nx, ny, nz in 8-bit RGB texture Building: -1 to 1  0 to 255 x in range -1 to 1 x+1 in range 0 to 2 (x+1)*0.5 in range 0 to 1 (x+1)*0.5*256 in range 0 to 256 min (255, floor ((x+1)*128)) in range 0 to 255 Accessing (via normalizing texture): x in range 0 to 255 is provided in range 0 to 1 x-0.5 in range -0.5 to +0.5(x-0.5)*2 in range -1 to 1 • Or directly in non-normalizing float RGB textures All numbers from 0.0 to 1.0 exclusive map to 0. Also want all numbers from 255.0 to 256.0 inclusive to map to 255

16. Relief Texture = Normal + Height Use 8-bit RGBA texture to encode nx, ny, nz, h Note: Some authors recommend that h representa normalized value in range 0 to 1 and that ascale factor be encoded in vertices.. to get small versus large bumps. relief texture

17. Relief Texture = Normal + Height OR Normal + Depth RGB = normal, A = depth (brighter = deeper)

18. Do we use a depth map or a height map? bottom HIGH bottom LOW top LOW top HIGH depth map(since top is 0) height map(since top is 1) Black is 0, White is 1

19. 95.4501 Tangent Space

20. Texture Coordinate Space (Or Tangent Space) In texture, x goes right, y goes up, z goes out, Artist Terminology v bitangent y x tangent u at least in a right-handed system w normal z

21. Transforming TO/FROM Tangent Space • Use the term vertex space to mean the space of the vertices; i.e., either Model space for geometry that can move. World space for static geometry. • The conventional notation/terminology: • T = [Tx, Ty, Tz] for the u-axis (tangent vector) B = [Bx, By, Bz] for the v-axis (bitangent vector) N = [Nx, Ny, Nz] for the w-axis (normal). all in vertex space

22. Transforming TO/FROM Tangent Space • Shaders need access to the averagetangent, bitangent, and normal at all vertices (pixels). • Need access to the triangles and texture coordinates to compute this. • In the discussion to follow, we’ll show how to compute this from one triangle… An average vector is computed by • Averaging the values of neighboring triangles, and • Normalizing the result (average changes the length)

23. Transforming TO/FROM Tangent Space • First, we compute T, B, N from an existing triangle. Then we build the following 3x3 transformations (right handed system). fromTangent= Tx TyTzto vertex Bx ByBzspace NxNyNz toTangent = T'xB'xNxfrom vertex T'yB'yNyspace T'zB'zNz almosttranspose(many use just the transpose) continued

24. Mapping To/From Tangent Space [Bx,By,Bz] [tx,ty,tz] = [x,y,z]* v-axis u-axis [Tx,Ty,Tz] [x,y,z] Tx TyTzBx ByBzNxNyNz • [x,y,z] = [tx,ty,tz]* fromTangentmapping T'xB'xNxT'yB'yNyT'zB'zNz toTangentmapping

25. Computing T, B, and N • We compute N from the triangle vertices using a cross product which is normalized; i.e., normalized (AxB) in a right handed system. • T and B are more complex. We start off giving you the answer... Then we provide the derivation which comes from the following. • B • A Mathematics for 3D Game Programming and Computer Graphics, Third Edition, Eric Lengyel, 2012, pp. 180-185.

26. Transforming TO/FROM Tangent Space • Given vertices P0, P1, P2 with texture coordinates [u0,v0], [u1,v1], [u2,v2], solve for T, B as follows: fromTangentSpace T= normalize ( t2Q1- t1Q2)B= normalize(-s2Q1 + s1Q2) Q1 = P1-P0, Q2 = P2-P0[s1,t1] = [u1-u0,v1-v0],[s2,t2] = [u2-u0,v2-v0], Compute Derivation coming up toTangentSpace T' = T -(N.T) B'=B- (N.B)N - (T'.B)T' and normalize

27. Before we Derive The Result • Need 2 capabilities • The inverse of a 2X2 transformation • The 3x3 rotation matrix that will map • Old axes [1,0.0], [0,1,0], [0,0,1] to new axes X-Axis, Y-Axis, Z-Axis(which for us will be T, B, N).

28. Inverse of a 2x2 Transformation -1 t2-t1 -s2s1 s1t1 s2t2 1 = s1t2 - s2t1 Proof t2-t1 -s2s1 s1t1 s2t2 1 * s1t2 - s2t1 1 s1 t2 - s2 t1 00 s1t2 - s2t1 = s1t2 - s2t1 1 00 1 =

29. What transformation will map standard axes to new Axes? • My current axes are [1,0,0], [0,1,0], [0,0,1]. • I want my new axes to be xAxis, yAxis, zAxis. • The transformation R that will do it is R = xAxisxxAxisyxAxiszyAxisxyAxisyyAxiszzAxisxzAxisyzAxisz Proof on next page.

30. Proof: Try Transforming [1,0,0]! Transformation R [1,0,0] * xAxisxxAxisyxAxisz yAxisxyAxisyyAxisz zAxisxzAxisyzAxisz original x-axis transformed x-axis [1,0,0] * R = [xAxisxxAxisyxAxisz] [0,1,0] * R = [yAxisxyAxisyyAxisz] [0,0,1] * R = [zAxisxzAxisyzAxisz]

31. Starting in Texture Space • In texture space, the u-,v-,w-axes are [1,0,0], [0,1,0], [0,0,1].We are defining T, B, N to be the corresponding vertex space vectors; i.e. Tx TyTzBx ByBzNxNyNz • T= [1,0,0]* fromTangentmapping This proves this matrix maps from texture space to vertex space. Tx TyTzBx ByBzNxNyNz • B= [0,1,0]* fromTangentmapping But we don’t YET know their actual numerical values Tx TyTzBx ByBzNxNyNz • N= [0,0,1]* fromTangentmapping

32. Deriving T and B From Vertex + Text. Coord. • In texture space, the u-,v-,w- axes are [1,0,0], [0,1,0], [0,0,1]. In vertex space, they are T, B, N. How do we compute T, B, N? P2 [u2,v2] bitangent B Q [u,v] a texture coordinate a vertex coordinate P0 [u0,v0] P1 [u1,v1] T tangent Q can be obtained from P0 by moving a little in the T direction and a little in the B direction N normal

33. Derivation From Texture Coordinates [u,v] go up along B • To solve for T and B (6 unknowns), plug in known values for Q [u,v]; i.e., P1 [u1,v1] and P2 [u2,v2]. P2 [u2,v2] go to the right along T Let Q be ANY point inside the triangle. Q [u,v] B T P0 [u0,v0] N P1 [u1,v1] Q-P0 = (u-u0)T + (v-v0)B directions only

34. Derivation P2 [u2,v2] B T Q [u,v] N P0 [u0,v0] P1 [u1,v1] Q-P0 = (u-u0)T + (v-v0)B P1-P0 = (u1-u0)T + (v1-v0)B Plug in P1 Q1 s1 t1 Q1 = s1T + t1B P2-P0 = (u2-u0)T + (v2-v0)B Plug in P2 Q2 s2 t2 Q2 = s2T + t2B

35. Derivation • Solve for T, B. How hard can it be? Q1 = s1T + t1B Q2 = s2T + t2B • In matrix form, s1Tx + t1Bxs1Ty + t1Bys1Tz + t1Bzs2Tx + t2Bxs2Ty + t2Bys2Tz + t2Bz Q1x Q1yQ1zQ2x Q2y Q2z = s1t1 s2t2 Tx TyTzBx ByBz = • Flipped Tx TyTzBx ByBz Q1x Q1yQ1zQ2x Q2y Q2z s1t1 s2t2 =

36. Derivation • So far Q1x Q1yQ1zQ2x Q2y Q2z -1 s1t1 s2t2 • Premultiply both sides by • Result -1 Tx TyTzBx ByBz Q1x Q1yQ1zQ2x Q2y Q2z s1t1 s2t2 = • More compactly Tx TyTzBx ByBz s1t1 s2t2 -1 T B Q1Q2 s1t1 s2t2 = =

37. Derivation • From -1 Q1Q2 s1t1 s2t2 = • Get t2-t1 -s2s1 Q1Q2 1 = s1t2 – s2t1 1 t2Q1- t1Q2-s2Q1 + s1Q2 = s1t2 – s2t1 T B T B T B

38. Notation • Short form 1 t2Q1- t1Q2-s2Q1 + s1Q2 = s1t2 – s2t1 • Is Equivalent to 1 (t2Q1 - t1Q2) T = s1t2 – s2t1 B (-s2Q1 + s1Q2) 1 = s1t2 – s2t1 T B

39. Summarizing T 1 t2Q1- t1Q2-s2Q1 + s1Q2 = and normalize B s1t2 – s2t1 where T = [Tx, Ty, Tz], B = [Bx, By, Bz] Q1 = P1-P0, Q2 = P2-P0[s1,t1] = [u1-u0,v1-v0],[s2,t2] = [u2-u0,v2-v0], fromTangent= Tx TyTzto vertex Bx ByBzspace NxNyNz WAIT: This is not as simple as it gets...

40. Summarizing T= normalize ( t2Q1- t1Q2)B= normalize(-s2Q1 + s1Q2) where Q1 = P1-P0, Q2 = P2-P0[s1,t1] = [u1-u0,v1-v0],[s2,t2] = [u2-u0,v2-v0], fromTangent= Tx TyTzto vertex Bx ByBzspace NxNyNz

41. Mapping from Vertex to Tangent Space • Mapping the other way is just the inverse; the inverse of a rotation is just the transpose… toTangent = T'xB'xNxfrom vertex T'yB'yNyspace T'zB'zNz fromTangent= Tx TyTzto vertex Bx ByBzspace NxNyNz = inverse of • That does not quite work for normals and tangents… Why not? • The transpose is not the inverse if theaxes are skewed (not perpendicular) • v-axis • w-axis • u-axis

42. So We Fix It (ASSUMES a normalized T and B) • Fix T by making it perpendicular to N. • T • T ' T' = T -(N.T)N • N shadow of A on B = |A|cos* (B/|B|) = |A||B|cos * (B/|B|2) = A.B * (B/|B|2) = (A.B) B if |B| = 1 A.B • Fix B by making it perpendicular to N and T ' (the fixed T). B'=B- (N.B)N - (T'.B)T' Also normalize T' andB' N fix T ' fix

43. Conclusion • Given vertices P0, P1, P2 with texture coordinates [u0,v0], [u1,v1], [u2,v2], solve for T, B as follows: fromTangentSpace Q1 = P1-P0, Q2 = P2-P0[s1,t1] = [u1-u0,v1-v0],[s2,t2] = [u2-u0,v2-v0], T= normalize ( t2Q1- t1Q2)B= normalize(-s2Q1 + s1Q2) Compute T' = T -(N.T)N B'=B- (N.B)N - (T'.B)T' toTangentSpace

44. 95.4501 Beginning Our Investigations

45. Background • Based on “ParallaxOcclusionMapping” demo by Tatarchuk from the June 2010 DirectX SDK. • Demo uses a 2-sided disc and builds tangent vectors using built-in DirectX routines. Look at Demo Demo “looks” good

46. Publication • Dynamic Parallax Occlusion Mapping with Approximate Soft Shadows, Natalya Tatarchuk, ATI Research, Inc, Siggraph 2005. with POM with Normal Mapping Other authors call this relief mapping

47. Desired Modifications • We wanted to be able to use a cube shape and move the demo to a local directory… • We created a texturedCube.x file but problems resulted.

48. Problems Encountered • File lookup does NOT use LOCAL MEDIA first (so we hard-wired the file path). • Parallax occlusion only worked on 2 of the 6 faces… Suspected the DirectX routines for computing tangent/bitangent vectors. Could not get it to work correctly for all faces. • Solution: We hand built the direct X mesh using vertex and index buffers and wrote our own routines. Demo (drawWilfCube (false for “texturedCube.x” and D3DXComputeTangentFrameEx; true for newVersion) + CubeForParallaxOcclusionMappingDemo.cpp) Real game engine building tools probably want their own routines anyway.

49. Further Changes • Wanted to do everything in a right handed system.D3DXMatrixLookAtRHD3DXMatrixPerspectiveFovRH • This screwed up the camera movement directions and the trackball for looking around… Host of low-level changes were needed to fix it… Kept it that way but in retrospect, might have been a mistake…

50. 95.4501 Changing the Original Shader Effect File