Motivations

1. A Hybrid Technique for Creating Meaningful Abstractions of Dynamic 3D Scenes in Real-TimeNiall Redmond and John DinglianaTrinity College Dublin

2. Motivations • Investigate whether multi-level NPR can be used as an effective tool in increasing user understanding and guidance in interactive scenes • Use non-photorealistic abstraction to render each scene object in a style suited to its importance • Using real-time NPR techniques • Meaningful abstractions which take 3D object information into account • Generic system which could be easily bound to a non-specific application

3. Background “Visual Interest and NPR: an Evaluation and Manifesto” Santella and DeCarlo 2002 “Real-time video abstraction”, Winnemoller, Olsen and Gooch, 2006. “Psychology and NPR: The Beginning of a Beautiful Relationship” Halper et al. 2003

4. False colour rendering • To create meaningful non-photorealistic renderings it is necessary to use the available 3D object information • False colour rendering technique • Two pass rendering system necessary • First pass renders each scene in a unique colour • Lighting, textures etc. disabled • Popular technique for object identification • Technique is fast enough to be implemented in real-time • Used to help find all edges and guide abstraction in our system

5. Kuwahara Filter • Digital processing of biomedical images, Kuwahara, Hachimura and Kinoshita, 1976 • Edge preserving blur filter • In each of the four regions the mean brightness and the variance are measured. The output value of the center pixel in the window is the mean value of that region that has the smallest variance.

6. Kuwahara Filter • Painterly effect • Suitable for real-time applications • Implemented with a pixel-shader • Good frame to frame coherence

7. Edge Detection • Object space edge detectors are reliable but extremely expensive • Image space edge detectors are necessary for finding edges within textures but can fail when finding edges between objects • We present a hybrid method which uses 3D scene information within an image space technique • Difference of Gaussian • Calculate the difference between two Gaussian blurred images with varied sigma values • Gaussian function: • Good compromise between speed and accuracy

8. Edge Detection • To find all scene edges reliably we run Difference of Gaussian edge detection on two images • False colour image to find object silhouette edges • Framebuffer image to find edges within textures • The two edge images can then be overlaid on the original image • Fast and effective technique

9. Various levels of abstraction • Kuwahara filter • Varying filter size • Iteratively applying filter • Edge detection • Vary edge thickness and boldness • Colour Variation • Vary colour brightness to increase abstraction

10. Object Segmentation • Highlight important scene content • False colour rendering used • Important scene objects are specified • Abstraction level for each object decided by the average pixel distance to the centre of the nearest important scene object • Allows for each scene object to be rendered in a style suitable to its importance

11. Adaptive Abstraction • Combining the different levels of NPR abstraction with the scene segmentation • General Algorithm • Render each object in a unique colour • Render the scene normally • Use framebuffer image and colour image as input to abstraction filter • Display resulting image adding colour fade • Possible Applications • Interactive applications • Highlighting popular sights or historical landmarks • Route finding

12. Performance • Implemented using pixel shaders • Speed of abstraction dependent on screen resolution not scene complexity • Tested on a Pentium D, 3.72 GHz computer with 2MB of RAM using a NVidia GeForce 7950 GX2 GPU • Virtual Dublin is a highly textured scene containing over 80,000 polygons • Highly interactive rates • 46 fps at 640x480 • 43 fps at 800x600 • 142fps at 1200x800

13. Evaluation • Automatic saliency metric used to confirm that our technique is successful in its goals • We used the algorithm presented by Laurence Itti et al in saliency-based visual attention for rapid scene analysis 1998 • Multi-scale approach for finding the location of salient regions based on local image structure • Saliency map created using regions of high contrast for colour, intensity and orientation • While user studies would provide more reliable information these preliminary tests are a good quick way of evaluating the system

14. Evaluation results • Tested on images both with and without abstraction • Results show that our methods are effective • Techniques can successfully increase saliency of scene objects The most salient point of an image found before abstraction (left images) and after abstraction (right images)

15. Future Work • Further work on multi-level abstraction techniques • Research user perception and judgement • Eye-tracking tests • Task based experiments • Test system on scenes with a variety of data types and scene sizes

16. Summary • Novel technique for generating NP renderings of 3D scenes in real-time • GPU pixel shaders used to ensure real-time performance • Hybrid approach using 3D scene information with image space techniques • Increases visual saliency of important scene objects • All objects rendered in an abstraction style suitable for their scene importance • Questions?