Practical Implementation of High Dynamic Range Rendering

1. Practical Implementation of High Dynamic Range Rendering Masaki Kawase BUNKASHA GAMES BUNKASHA PUBLISHING CO.,LTD http://www.bunkasha-games.com http://www.daionet.gr.jp/~masa/

2. Agenda • What can be done with HDR? • Dynamic Range • Implementation on DX8 hardware • Implementation on DX9 hardware • Multiple Gaussian Filters • HDR in games • References

3. What can be done with HDR? • Dazzling light • Dazzling reflection • Fresnel reflection • bright reflection in the normal direction • Exposure control effects • Realistic depth-of-field effects • Realistic motion blur

4. Dazzling Light

5. Dazzling Reflection

6. HDR Fresnel Bright reflection off low-reflectance surfaces

7. Exposure Control

8. HDR Depth-of-Field Future perspective

9. HDR Motion Blur Future perspective

10. Dynamic Range • The ratio of the greatest value to the smallest value that can be represented • Displayable image • 28 Low dynamic range (LDR) • Frame buffer of absolute luminance • Render the scene in absolute luminance space • >232 represents all luminances directly • Frame buffer of relative luminance • Apply exposure scaling during rendering • >215~16 dark regions are not important

11. HDR Frame Buffers • For glare generation • When rendering with relative luminances: • Ideally, more than 215~16 • In games • 212~13 (4,000~10,000) is acceptable

12. HDR Environment Maps • Very important for representing: • Realistic specular reflection • Dazzling specular reflection • Specular reflectance of nonmetals • Reflectance in the normal direction is typicallyless than 4% • Bright light remains bright after such low reflection • To maintain dazzles after reflection of ~1-4% • Dynamic range of more than 10,000 or 20,000 is necessary

13. Implementation on DX8 Hardware • We have no choices • Pixel Shader 1.x • Integer operations only • HDR buffer formats • Low-precision buffers only • Use the alpha channel as luminance information • Fake it to achieve believable appearance • Accurate calculation is not feasible

14. Glossy reflection material Fake HDR Pixel Shader ps_1_1 text0 text1 text2 madr0.rgb, v0, t2, v1 // Scale the primary diffuse color by // shadow/light map, and add the result of // other per-vertex lighting +mult0.a, v1.a, t0.a // Scale the specular reflectance // by gloss map mulr0.rgb, t0, r0 // Modulate diffuse color with decal texture +mulr0.a, t0.a, t1.a // r0.a = specular reflectance * envmap luminance mult1.rgb, t1, c0 // Modulate envmap with specular color +mulr1.a, r0.a, t1.a // Envmap brightness parameter // r1.a = specular reflectance * envmap luminance * gloss map lrpr0.rgb, t0.a, t1, r0 // Reflect the envmap by specular reflectance mulr1.a, r1.a, c0.a // Envmap brightness parameter // r1.a = specular reflectance * envmap luminance * gloss map * Clamp(gloss * 2, 0, 1) lrpr0.rgb, r1.a, t1, r0 // Output color // Interpolate the envmap color and the // result of LDR computation, based on // the envmap brightness parameter (r1.a) +lrpr0.a, r1.a, t1.a, r0.a // Output luminance information // v0.rgb : Diffuse color of primary light // v1.rgb : Color for other lights/ambient // pre-scaled by (exposure * 0.5) // // v1.a : Specular reflectance (Fresnel) // // t0.rgb : Decal texture (for diffuse) // t0.a : Gloss map (for specular) // t1.rgb : Envmap color // t1.a : Envmap luminance // t2.rgb : Shadow/light map // // c0.rgb : Specular color // c0.a : Clamp(gloss * 2, 0, 1)

15. Generating Displayable Image • Extract high-luminance regions Threshold : ~0.4-0.5 • Generate glare • Reference: • Kawase, Masaki, “Frame Buffer Postprocessing Effects in DOUBLE-S.T.E.A.L (Wreckless)” • Generate a displayable image • Calculate the luminance from the frame buffer • Add the result of glare generation to the luminance

16. Notes on DX8 Implementation • Accurate calculation is not feasible • How to make it believable by faking • Based on appearance rather than theory

17. Implementation on DX9 Hardware • There are currently many limitations • Choose implementations accordingly • Pixel Shader • Pixel Shader 2.0 or later • Pixel Shader 1.x • Buffer formats for HDR • High-precision integer/float buffers • Low-precision integer buffers

18. Issues with High-Precision Buffers • Memory usage • At least twice as much memory as the conventional full-color buffer is needed • Limitations • Alpha blending cannot be used • Texture filtering cannot be used with floating-point formats • Some systems don’t support them • The situation is not good…

19. Use Low-Precision Buffers • Make use of low-precision buffers • A8R8G8B8 / A2R10G10B10 etc. • Low memory consumption • Alpha blending can be used

20. Compression with Tone Mapping • Render directly to displayable format • Nonlinear color compression • Effectively wide dynamic range • Reference: • Reinhard, Erik, Mike Stark, Peter Shirley, and Jim Ferwerda, “Photographic Tone Reproduction for Digital Images” • The alpha channel is not used • Can be used for any other purpose

21. Environment Map Formats • Relatively low resolution • Alpha channel/blending is not very important • Use the 16-bit integer format if enough memory storage is available • Treat it as having an interval of [0, 256] or [0, 512] • Texture filtering can be used • In the future • Do it all with A16B16G16R16F

22. Low-Precision Environment Maps • Use them when: • High-precision buffers are not supported, or • Memory storage is limited • If the fill-rate of your system is relatively low • Use the same format as used in DX8 fake HDR • If the fill-rate is high enough: • Nonlinear color compression • Similar to tone mapping • Store exponents into the alpha channel • More accurate operations are possible • Using it just as a scale factor is not enough • Even the DX8 fake HDR has a much bigger impact

23. Color Compression • Similar to tone mapping • Encode when rendering to an environment map Offset : luminance curve controlling factor (~2-4) • A bigger offset means: • High-luminance regions have higher resolutions • Low-luminance regions have Lower resolutions • Decode when rendering to a frame buffer • From the environment map fetched d : a small value to avoid divide-by-zero

24. Color Compression • Use carefully • Mach banding may become noticeable on reflections of large area light sources • e.g. Light sky

25. E8R8G8B8 • Store a common exponent for RGB into the alpha channel • Use a base of 1.04 to 1.08 offset : ~64-128 • Base=1.04 means dynamic range of ~23,000 (1.04256) • A bigger base value means: • Higher dynamic range • Lower resolution (Mach banding becomes noticeable)

26. E8R8G8B8 Encoding (HLSL) // a^n = b #define LOG(a, b) ( log((b)) / log((a)) ) #define EXP_BASE (1.06) #define EXP_OFFSET (128.0) // Pixel Shader (6 instruction slots) // rgb already exposure-scaled float4 EncodeHDR_RGB_RGBE8(in float3 rgb) { // Compute a common exponent float fLen = dot(rgb.rgb, 1.0) ; float fExp = LOG(EXP_BASE, fLen) ; float4 ret ; ret.a = (fExp + EXP_OFFSET) / 256 ; ret.rgb = rgb / fLen ; return ret ; } // More accurate encoding #define EXP_BASE (1.04) #define EXP_OFFSET (64.0) // Pixel Shader (13 instruction slots) float4 EncodeHDR_RGB_RGBE8(infloat3 rgb) { float4 ret ; // Compute a common exponent // based on the brightest color channel float fLen = max(rgb.r, rgb.g) ; fLen = max(fLen, rgb.b) ; float fExp = floor( LOG(EXP_BASE, fLen) ) ; float4 ret ; ret.a = clamp( (fExp + EXP_OFFSET) / 256, 0.0, 1.0 ) ; ret.rgb = rgb / pow(EXP_BASE, ret.a * 256 - EXP_OFFSET) ; return ret ; }

27. // Pixel Shader (5 instruction slots) float3 DecodeHDR_RGBE8_RGB(infloat4 rgbe) { float fExp = rgbe.a * 256 - EXP_OFFSET ; float fScale = pow(EXP_BASE, fExp) ; return (rgbe.rgb * fScaler) ; } Encoding/decoding should be done using partial-precision instructions Rounding errors inherent in the texture format are much bigger E8R8G8B8 Decoding // If R16F texture format is available, // you can use texture to convert alpha to scale factor float3 DecodeHDR_RGBE8_RGB(infloat4 rgbe) { // samp1D_Exp: 1D float texture of 256x1 // pow(EXP_BASE, uCoord * 256 - EXP_OFFSET) float fScale = tex1D(samp1D_Exp, rgbe.a).r ; return (rgbe.rgb * fScale) ; }

28. Glossy reflection material Rendering with Tone Mapping float4 PS_GlossReflect(PS_INPUT_GlossReflect vIn) : COLOR0 { float4 vDecalMap = tex2D(samp2D_Decal, vIn.tcDecal) ; float3 vLightMap = tex2D(samp2D_LightMap, vIn.tcLightMap) ; float3 vDiffuse = vIn.cPrimaryDiffuse * vLightMap + vIn.cOtherDiffuse ; vDiffuse *= vDecalMap ; // HDR-decoding of environment map float3 vSpecular = DecodeHDR_RGBE8_RGB( texCUBE(sampCUBE_EnvMap, vIn.tcReflect) ) ; float3 vRoughSpecular = texCUBE(sampCUBE_DullEnvMap, vIn.tcReflect) ; float fReflectance = tex2D( samp2D_Fresnel, vIn.tcFresnel ).a ; fReflectance *= vDecalMap.a ; vSpecular = lerp(vSpecular, vRoughSpecular, fShininess) ; float3 vLum = lerp(vDiffuse, vSpecular, fReflectance) ; // HDR tone-mapping encoding float4 vOut ; vOut.rgb = vLum / (vLum + 1.0) ; vOut.a = 0.0 ; return vOut ; } struct PS_INPUT_GlossReflect { float2 tcDecal : TEXCOORD0 ; float3 tcReflect : TEXCOORD1 ; float2 tcLightMap : TEXCOORD2 ; float2 tcFresnel : TEXCOORD3 ; // Exposure-scaled lighting results // Use TEXCOORD to avoid clamping float3 cPrimaryDiffuse : TEXCOORD6 ; float3 cOtherDiffuse : TEXCOORD7 ; } ;

29. Generating Displayable Image • Extract high-luminance regions Threshold : ~0.5-0.8 • Divide by (1 - Threshold) to normalize • Generate glare • Use an integer buffer to apply texture filtering • Hopefully, a float buffer with filtering… • Generate a displayable image • Add the glare to the frame buffer

30. Notes on DX9 Implementation • High-precision buffers • Consumes a lot of memory • No blending capability • Low-precision buffers • Pixel shaders are expensive • Consider fake techniques like DX8 • High performance • Low memory consumption • Very effective

31. Multiple Gaussian Filters • Bloom generation • A single Gaussian filter does not give very good results • Small effective radius • Not sharp enough around the light position • Composite multiple Gaussian filters • Use Gaussian filters of different radii • Larger but sharper glare becomes possible

32. Multiple Gaussian Filters

33. Multiple Gaussian Filters Original image

34. Multiple Gaussian Filters • A filter of large radius is very expensive • Make use of downscaled buffers • A large radius means a strong low-pass filter • Apply a blur filter to a low-res version of the image and magnify it by bilinear filtering  The error is unnoticeable • Change the image resolution rather than the filter radius • 1/4 x 1/4 (1/16 the cost) • 1/8 x 1/8 (1/64 the cost) • 1/16 x 1/16 (1/256 the cost) • 1/32 x 1/32 (1/1024 the cost) • … • Even a large filter of several hundred pixels square can be applied very quickly

35. Applying Gaussian Filters to Downscaled Buffers 1/4 x 1/4 (256x192 pixels) 1/8 x 1/8 (128x96 pixels) 1/16 x 1/16 (64x48 pixels) 1/32 x 1/32 (32x48 pixels) 1/64 x 1/64 (16x12 pixels)

36. Bilinear Filtering and Composition • Magnify them using bilinear filtering and composite the results • The error is almost unrecognizable + + + =

37. Good Enough!

38. Notes on Filter • Use high-precision formats for low-res buffers • Don’t take too many samples • Very expensive, especially for high-res images

39. HDR in Games • Should be appealing rather than accurate • Accuracy is not important for players • Even an inaccurate scene can be appealing • Cost and performance • The key is to understand the effects of HDR • Devise a fake technique that is fast enough and produces believable results • Accurate HDR rendering is still hard in games • Use HDR only for effects that give a large impact • Use sprites if you want to generate glare only for the sun, which is much faster and gives high quality results

40. HDR in Games • Integer formats have very limited dynamic range • Render in relative luminance space • Apply exposure scaling during rendering • High precision is not needed for the dark regions after exposure scaling • As those regions remain dark in the final image • Carefully choose the range to maximize effective precision

41. HDR in Games • Future perspective • All operations can be done in float • Effective use of HDR • Depth-of-field with the shape of aperture stop • Motion blur with high luminances not clamped

42. References • Reinhard, Erik, Mike Stark, Peter Shirley, and Jim Ferwerda, “Photographic Tone Reproduction for Digital Images” • Mitchell, Jason L., “Real-Time 3D Scene Post-Processing” • DirectX 9.0 SDK Summer 2003 Update, “HDRLighting Sample” • Debevec, Paul E., “Paul Debevec Home Page” • Kawase, Masaki, “Frame Buffer Postprocessing Effects in DOUBLE-S.T.E.A.L (Wreckless)”