Animating Chinese Landscape Paintings and Panoramas

1. Animating Chinese Landscape Paintings and Panoramas By Nelson Siu-Hang CHU Department of Computer Science The Hong Kong University of Science and Technology

2. Outline • Introduction • Motivations • Two IBMR Approaches • Multi-Perspective Modeling • Baseline Extrusion • An Improved Graphical User Interface • Conclusions Computer Science Department Hong Kong University of Science and Technology

3. Introduction • Image Based Modeling & Rendering (IBMR) • Pre-acquired images -> novel views Computer Science Department Hong Kong University of Science and Technology

4. Introduction (cont.) • Image Based Modeling & Rendering (IBMR) • Advantages: • Realism with less computation • Computation independent of scene complexity • Disadvantage: • Number of images needed is large, in general • Success in film industry • e.g. The Matrix [SoftImage Magazine Aug. ’99] Computer Science Department Hong Kong University of Science and Technology

5. Introduction (cont.) • Tour Into the Picture (TIP) [HAA97] • “Visually 3D” animation from a single image • Easy-to-use graphical user interface (GUI) • Good results in relatively short time • Works especially well for inputs of paintings Computer Science Department Hong Kong University of Science and Technology

6. Motivation • Impressive results using the TIP technique TIP By Hoi Ng 1982 Computer Science Department Hong Kong University of Science and Technology

7. Motivation (cont.) • Chinese landscape paintings • Horizontal or vertical scrolls from Mustard Seed Garden Painting Manual, revised version by Ch’ao Hsun, 1887 Computer Science Department Hong Kong University of Science and Technology

8. Multiple Perspectives in Chinese Landscape Paintings Eye locations Computer Science Department Hong Kong University of Science and Technology

9. Motivation (cont.) • Fixed-point perspective • As in photographs or western paintings • At most a spherical panorama • Space depicted is limited • Moving-point perspective • As in Chinese landscape painting or multi-perspective images e.g. in [WFH+97] • Space NOT limited • Goal: Model and animate multi-perspective images Computer Science Department Hong Kong University of Science and Technology

10. Motivation (cont.) • Panorama • Compact representation • Easy to produce • Cost effective for virtual reality (VR) Computer Science Department Hong Kong University of Science and Technology

11. Motivation (cont.) • Panorama Viewer • Camera pan, zoom, tilt only • Fixed viewpoint – we can’t move • VR techniques giving better VR experience exist, but too costly: • Special hardware • Large input data So, it’s valuable if can enhance immersive effect of panorama without much increase in production and computational costs Computer Science Department Hong Kong University of Science and Technology

12. Outline • Introduction • Motivations • Two IBMR Approaches • Multi-Perspective Modeling • Baseline Extrusion • An Improved Graphical User Interface • Conclusions Computer Science Department Hong Kong University of Science and Technology

13. The TIP Technique Computer Science Department Hong Kong University of Science and Technology

14. Sub-scene 1 Sub-scene 2 First Approach: Multi-Perspective Modeling (MPM) • Solution: introduce local and global models; switch them smoothly • TIP is not applicable for multi-perspective images: • horizons at different levels • multiple cavities Computer Science Department Hong Kong University of Science and Technology

15. Rectangle • Cylinder • Sphere MPM: Global Modeling • Global model Computer Science Department Hong Kong University of Science and Technology

16. A box model as in TIP • Foreground objects as “billboards” MPM: Sub-scene Modeling • Local model for a sub-scene Computer Science Department Hong Kong University of Science and Technology

17. Pre-processing Digitize the input image Modeling Model back-ground using TIP interface Identify sub-scenes Prepare: Foreground mask Background image Select a global model Build billboards for foreground objects from mask Rendering Record animation key-frames Generate smooth camera path Render & output animation MPM: System Overview For each sub-scene Computer Science Department Hong Kong University of Science and Technology

18. MPM: Model Switching Scene entry points Computer Science Department Hong Kong University of Science and Technology

19. Panorama as Input Sub-scene 2 Sub-scene 1 Computer Science Department Hong Kong University of Science and Technology

20. MPM: Model Switching Computer Science Department Hong Kong University of Science and Technology

21. Animation Results • Painting “Fishing village in mild snow” • Real scene “HKUST Piazza” Computer Science Department Hong Kong University of Science and Technology

22. Outline • Introduction • Motivations • Two IBMR Approaches • Multi-Perspective Modeling • Baseline Extrusion • An Improved Graphical User Interface • Conclusions Computer Science Department Hong Kong University of Science and Technology

23. Second Approach: Baseline Extrusion • Targets at panoramic inputs • Builds a unified model • Assumptions • A fixed viewpoint (panorama) • A flat ground is present in the scene Computer Science Department Hong Kong University of Science and Technology

24. Baseline Second Approach: Baseline Extrusion • Main Idea: • user draws “baselines” on perspective views of the input image to indicate positions and orientations of billboards • Extrude the baselines to form billboards Computer Science Department Hong Kong University of Science and Technology

25. Second Approach: Baseline Extrusion • Basically two types: • Indoor: man-made structures • Outdoor: usually a sky is captured Computer Science Department Hong Kong University of Science and Technology

26. Longitude and latitude coordinates • Input image represented as a spherical surface, with polar coordinates (, ) Radius=1  o  Computer Science Department Hong Kong University of Science and Technology

27. By considering OBV’ and OAV, Calculation of billboard positions from baseline Computer Science Department Hong Kong University of Science and Technology

28. Sample structure inferred from a baseline Computer Science Department Hong Kong University of Science and Technology

29. Baseline Baseline Extrusion • Billboard heights will be determined by: • Scanning mask image, or • Ceiling height Computer Science Department Hong Kong University of Science and Technology

30. Specific model component: Ceiling Plane • Define the ceiling height interactively: Computer Science Department Hong Kong University of Science and Technology

31. Animation Result • Real scene “Chapel Quad” Input panorama Computer Science Department Hong Kong University of Science and Technology

32. Outline • Introduction • Motivations • Two IBMR Approaches • Multi-Perspective Modeling • Baseline Extrusion • An Improved Graphical User Interface • Conclusions Computer Science Department Hong Kong University of Science and Technology

33. Perspective Grids Graphical User Interface(GUI) Improvement: Perspective Grids In TIP, a good GUI is the key! Spidery mesh Computer Science Department Hong Kong University of Science and Technology

34. 45° Deriving model from a spidery mesh Computer Science Department Hong Kong University of Science and Technology

35. 45-degree vanishing point Straight-ahead vanishing point Horizon Drawing Perspective Grids Computer Science Department Hong Kong University of Science and Technology

36. Perspective Grids: User operations Computer Science Department Hong Kong University of Science and Technology

37. Advantages of perspective grids • Visual modeling • Show a better sense of depth • Intuitively estimate the shape of the model • Tool for non-technical users – knowledge of field-of-view or focal length not required! • Draw the grids using 2D geometry • Real-time feedback guaranteed Computer Science Department Hong Kong University of Science and Technology

38. Conclusions • Proposed two approaches for producing animations • From a single image with multiple perspectives • From a panorama • As extensions of TIP • Tour into images of higher spatial complexity • Produce longer animations to make it more entertaining • As extensions of panorama viewing • Produce animations of higher immersive effect • Reasonable cost Computer Science Department Hong Kong University of Science and Technology

39. Conclusions (cont.) • Limitations of our IBMR approaches • Navigation limited by geometry and texture information captured by the input image • Multi-Perspective Modeling • Some constrained nodes in animation path • Not suitable for free navigation • Baseline Extrusion • Currently vertical billboards only • Modeling inclined surfaces & complex objects left as future work • Proposed a minor GUI improvement Computer Science Department Hong Kong University of Science and Technology

40. Questions?