Using wavelets on the XBOX360

1. Using wavelets on the XBOX360 Mike Boulton Senior Software Engineer Rare/MGS mboulton@microsoft.com For current and future games San Francisco, GDC 2008

2. Introduction • Fixed compression methods such as DXT are becoming antiquated • Distribution of data “density” becoming less uniform as complexity increases • Why can’t compression efforts be focused where it matters most? • Wavelets can help!

3. What are wavelets? • Wavelets are mathematical functions formed from scaled and translated copies of a few basis functions • The advantage is their ability to localize functions in both frequency and space • Fourier just frequency • “Point basis” just space • There are lots of common wavelet bases

4. Why 2D Haar? • This talk will focus on 2D Haar wavelets • Haar is the simplest basis set • In general, this means more basis terms and/or blocky artefacts • But also easiest to implement  • “Blocky” nature of bases is less of a problem for operations such as integration • What do the wavelets look like?

5. 2D Haar continued... • Three basis wavelet functions, plus ‘solid’ scaling function • White represents +1, black -1

6. The wavelet advantage • Local coverage allows windowed changes • Compared with spherical harmonics, which have global cover • This means that local changes only involve local bases • Mip-map chains not required for image decompression • Truly variable compression • Can focus efforts where it counts

7. Usage examples • Real-time shader image decompression • Lighting • Static shadow maps • Displacement maps over large areas • Easy dynamic texture packing • Geometry representations • ...many more!

8. Talk focus • Real-time shader image decompression • Real-time on XBOX360 GPU • 500Hz+ for full-screen 720p monochrome decompression • Double-product integration for relighting • Real-time (ish!) on XBOX360 GPU • Other applications • E.G. geometry representation • If time!

9. Real-time image decompression • Image broken up into 16x16 texel blocks • Each block compressed into wavelet sub-tree • Texture at 1/16th resolution stores scaling coefficient and offset to start of sub-tree • Why? • Decompression performance should be independent of main image resolution • Each sub-tree fits inside a texture cache tile, so no traversal cache thrashing • Can unroll traversal loop in shader for perf. gain

10. Real-time image decompression continued... • Given a (u,v) coordinate, pixel shader traverses appropriate sub-tree for final value • Mip-map chain not required • Obtained as by-product of traversal • Intermediate memory not required • All decompression performed directly in the pixel shader • Dynamic predication works well • Good vector coherency if minification avoided

11. Real-time image decompression continued... • How is the wavelet tree represented? • Stored breadth-first in a line texture • Multiple ways to pack the data • Example: ARGB8 • {r, g, b} stores windowed wavelet basis coefficients • {a} stores linear offset to child of node, if one exists • If no child exists, we jump to an “empty” node • Wait for the rest of the pixel vector to finish • Can use wider data formats • E.g. U16, F32

12. Real-time image decompression continued... Uncompressed with mipmaps ~2MB @ 1578fps

13. Real-time image decompression continued... Wavelet compressed, cut-off 0.05, 249KB @ 579fps

14. Real-time image decompression continued... Wavelet compressed, cut-off 0.08, 157KB @ 622fps

15. Real-time image decompression continued... • Notice how areas of high contrast have their detail preserved, and areas of lower contrast are “smoothed” out • Can use a different notion of importance!

16. Real-time image decompression continued... • Has advantages over fixed DXT-like compression schemes: • Can be lossless in areas you need it to be • Particularly important for “data” textures, such as SH coefficients • Will do a much better job for images with areas of low contrast • Again, higher-order SH terms are a good example • But most surface-parameterised data is likely to be like this • Can use “focus” ability to increase the area of parameterisation

17. Real-time image decompression continued... • (Video)

18. Double-product integration • Another good application is relighting • Work by [Ng, Ramamoorthi and Hanrahan] • Diffuse BRDF • We represent both the transfer function (with cosine) and the environment as wavelet trees in a texture

19. Double-product integration continued... • Here we need to traverse only the intersection of both trees • So if one is simple, should get good performance • Parallel GPU traversal needs to know how to “jump over” bits of either tree not contained in the intersection • If a node has a child, store linear offset to sibling or ancestor (could be root) • Then we can jump over whole child branch if the other tree has no children at that point

20. Double-product integration continued... • Performance is a function of the size of the intersection between both trees • Plus other factors such as cache behaviour • Loads of other interesting ideas • Transfer function wavelet trees have a tendency to be quite similar in localised patches • Clustering scheme would help further • Could store “representative” tree for each patch, and then a (smaller) “residual” tree per texel of patch

21. Double-product integration continued... • (Video)

22. Other applications • Wavelets can be used to define surface deformations • As a “dynamic” displacement map • Can keep memory overhead constant by removing oldest high-frequency terms for new deformation terms • Windowed nature of wavelets makes applying localised deformations simple • Can easily modify e.g. moment of inertia