Real-time Video Effects Using Programmable Graphics Cards

1. Real-time Video Effects Using Programmable Graphics Cards Master of Science Thesis Klas Skogmar klas@skogmar.com Klas Skogmar, Lund Institute of Technology

2. Introduction • Graphics cards have much computing power but are only used by 3D applications • Video and image editing programs often needs to perform per pixel operations that are ideal for graphics cards Klas Skogmar, Lund Institute of Technology

3. Problems • How can graphics cards be used for displaying and altering video? • What kinds of effects are suitable for a 3D environment? • Is it possible to speed up effects by using the graphics card? • How are the transfer rates affecting the system? • How much disc space is required and how does this affect the system? Klas Skogmar, Lund Institute of Technology

4. Programmable graphics cards • Graphics cards can do massive parallel calculations • 3D developers required more flexibility • 3D card producers introduced “Vertex shaders” and “Pixel shaders” • Makes it possible to program graphics cards in a similar way to processors Klas Skogmar, Lund Institute of Technology

5. Effects on graphics cards • Color correction • Masking • Color keying • Compositing • Transitions • Painting • 3D effects Klas Skogmar, Lund Institute of Technology

6. High resolution challenges HDTV standard 1080p requires throughput of 156 Mb/sec uncompressed. Current broadcasts takes approximately 30 Mb/sec uncompressed Challenges: • Disc space • Transfer speed • Other problems (e.g. dynamic range) Klas Skogmar, Lund Institute of Technology

7. Working practice • Real-time • Expensive • Scanning film • Many formats • High resolution • Proxies and AOI • Specialized hardware Klas Skogmar, Lund Institute of Technology

8. Available technologies • Dedicated hardware • OpenGL (Imaging subset) • DirectX (Direct3D and DirectShow) • Quicktime • ATI’s and Nvidia’s Klas Skogmar, Lund Institute of Technology

9. DirectShow • Microsoft’s media development environment • Built up of filters – each with its own task • Filters are assembled in a filter graph • Is supported by many video and image formats Klas Skogmar, Lund Institute of Technology

10. My program • Built using MFC • Uses a DirectShow filter • Shows how DirectShow can be used Klas Skogmar, Lund Institute of Technology

11. Performance • My program was slow • ATI has developed a program that is fast (uses VMR-9) • Serious Magic’s Texture Download Benchmark • Matt Craighead’s program Klas Skogmar, Lund Institute of Technology

12. Performance Results: • ATI’s program shows the potential – adding effects do not give visible frame drops • The other programs shows the transfer speed from the graphics card to the processor – it is not enough, but driver updates enhances the performance Klas Skogmar, Lund Institute of Technology

13. Conclusions • It is already a technique that can be used! • Drivers has to be enhanced to allow the modifications to be written back to memory • Graphics cards are more flexible than dedicated hardware • Effects have to be rewritten • Many effects are more suitable to do using graphics cards Klas Skogmar, Lund Institute of Technology

14. Conclusions • The transfer to and from disc can be a problem • There is a need for modules (e.g. DirectShow filters), that makes it easy to use programmable graphics cards Klas Skogmar, Lund Institute of Technology