Cinematography in Augmented Reality

1. Cinematography in Augmented Reality Richard Shemaka

2. Exploring the notion of camera and presentation in augmented reality • Background on Camera and Cinematography • Introduction to AR • Camera in AR • Two Approaches to Cinematography in AR

3. Camera and Cinematography • Ever since the advent of film, the notion of camera has become an import tool for media makers to convey their ideas • The moving photographic image gave rise to a new language of emotion and meaning-making: cinematography • Some larger considerations of cinematography include lighting, camera movement, composition (framing, blocking, etc), and special effects

4. Camera in Entertainment Media • While traditional cinematography is concerned with the photographic image, we can derive important aspects with respect to any moving image • The lessons learned from photography and cinematography have been applied to great affect in new media such as websites and video games Uncharted 2: Among Thieves Naughty Dog - 2009

5. Augmented Reality • An emerging media which blends reality with computer images • First appeared in the early 1990’s • In recent years, advances in mobile technologies have allowed for more wide-spread development

6. History of Camera in AR • Wearable AR systems focused on first-person presentation of material • Focus was more on the presentation of information • Ease-of-use and transparency have been the goals of many AR camera systems

7. Camera in AR • Most AR cameras rely on user controlled position and orientation • This alone renders many of the traditional techniques of cinematography irrelevant • E.g. calculated camera movement

8. Leveraging Cinematography in AR • As with most new forms of media, we can bring our knowledge of older media to bear when designing new experiences (remediation) • Recently, video games have started to integrate cinematic camerawork with interactive controls • Because most interactive AR applications rely on a user-controller camera, new techniques and approaches will be needed

9. Lighting • Lighting is a powerful cinematic tool that could be easily adapted to AR • Both scene lighting and interactive light-based mechanics could contribute to a scene’s presentation

10. Framing and Composition • Framing and composition in film are generally accomplished with careful camera placement • I am proposing two separate approaches to accomplishing these effects in a typical handheld augmented reality interface • Guiding the user’s gaze within the space • Adjusting the scene according to the user’s view

11. Two Separate Approaches Focus of Attention Camera-Aware Scenes The scene recognizes and adapts to the way it is being presented Props and characters change themselves to better ‘fit the shot’ • The user’s attention is directed towards particular parts of the scene • Borrows directly from film with techniques such as focus, action, framing, etc.

12. Focus of Attention • Loaded Scenes • In this approach (originally taken from stage), the scene is constructed in such a way to lead the user towards a particular view • Borrows heavily from composition • Action • Actors and events direct the user’s attention towards something particular

13. Camera-Aware Scenes • Actors’ positioning and pose • Having the actors in a shot change to better present themselves • Shifting scene • The scene shapes itself based on the way it is viewed

14. Implementation • In order to explore these designs in practice, I plan to build a suite of AR prototypes which focus on cinematic presentation • With the nature of prototyping, I will not be going into these with any expectation beyond leveraging the designs I am outlining for the project

15. Platform • Qualcomm’s AR SDK • Natural feature tracking • API for some AR-specific mechanics • Unity3D for Android • Game engine + IDE • Integrated with Qualcomm’s SDK