Levels of Realism: From Virtual Reality to Real Virtuality - Perception and Virtual Time

1. Levels of Realism: From Virtual Reality to Real Virtuality -Perception and Virtual Time Alan Chalmers† Andrej Ferko‡ † Warwick Digital Laboratory, UK ‡ Comenius University, Slovakia

2. Courtesy of recomCGI and Steffen Schraegle, produced via SpheronVR Technology “Holy Grail” Realism in Real-Time • Real-Time ≥ 25 fps • Realism …?

3. Introduction Virtual reality 1960’s Morton Heilig’s Sensorama Ivan Sutherland’s head mounted display Modern multi-modal VR systems CAVE’s, data-gloves, etc. Needs real-time Trade off realism to achieve this GPUs make things faster Not physically based

4. Types of realism 1998: Lengyel’s rendering spectrum Images which are appearance based Geometry which is physically based 2001: Ferwerda Physical realism Visual simulation Photorealism Visual response Functional realism Visual information

5. Physical realism Exact match Spectral irradiance values Match at all points within the real and virtual scene Scene modelled precisely Geometry Materials Light transport BUT All include approximations Especially displays (even HDR is only 3,000 cd/m2)

6. Photorealism Resultant rendered image Indistinguishable from photograph of real scene Other forms Lumigraph [Gortler et al. 1996] Image based lighting [Debevec et al. 1996] But Make rendered images more like a photograph Blurring Increasing Gaussian noise

7. Being there

8. Functional realism Image provides Sufficient information to enable viewer to complete task Correct relative size and shape But not material properties

9. Levels of realism One-to-one mapping Virtual world to Real world experience Adopt same learning strategy Real and virtual worlds High confidence in results

10. Believable realism Believable Realism!

11. Scruffy textures

12. Classification Games Photographs Real virtuality Vermeer Is Believable IB Photorealism Virtual Reality IBNP IBIP NBNP NBIP Cartoons NPR Avatars Uncanny valley Not Believable NB Abstract art Functional realism Not Physics NP Is Physics IP

13. Comparing real and virtual scenes Physical test environment with diffuse materials [Meyer et al. 1986] Comparing photograph with real scene [Rushmeier et al. 1995] Visual Difference Predictor (VDP) [Daly 1993], [Myszkowski 1998], [Mantiuk et al. 2005] Karol’s new method Myszkowski et al. @ SIGGRAPH 2008

14. Holistic approach Three different human visions Monocular, binocular, and ambient Four parallel processes Head, eye, visual perception, cognitive processes Visual equivalence [Ramanarayanan et al. 2007] Judgement of lightness [McNamara et al. 2000]

15. Multi-modalities Human perceives environment with all senses Sight Sound Smell Touch Taste Crossmodal effects can be considerable

16. Presence Typically a measure of technical immersion Higher quality  more immersion But Immersion in books and films Can quantify user’s engagement with VE Not a measure of realism

17. Preconditioning Human imagination Dungeons & Dragons Virtual Vietnam Habituation Familiarity may make user perceive less

18. Nature of task Top down process Attention under volitional control What we perceive is strongly dependent on the task Looking for a street sign Finding a coffee shop On patrol in (hostile) environment

19. There-reality Same perceptual response as if “there” in real world Exploits limits of human perception Only render in high quality what user is attending to Selective rendering Benefits from crossmodal interactions Can save significant computational effort Experience Active – infinitive  limited by technology Passive

20. The perception equation Function of task (t) and preconditioning (ρ): P(t, ρ) = ωv V + ωa A + ωs S + ωt T + ωf F V=Visuals, A=Audio, S=Smell, T=Taste, F=Feel Σωi = 1 ωiare threshold values Above threshold there is no perceptual difference

21. Example Preconditioning Experienced sailor Sound of water across hull Feel of wind on skin

22. Other approaches Eg. Neurolinguistic programming (NLP) Sensory input channels Visual Audio Kinesthetic Olfactory Gustatory Can be used to represent eg. desire, emotions … Initially: Interpersonal communication model Now: Business and management training

23. Virtual time issue Perceptual equation re-read… Sensory input channels in real and virtual time (Borges, 2 past) Visual Audio Kinesthetic Olfactory Gustatory Higher level of perception… VR as a semiotic system How to define and measure interestingness? Up to now: data mining, Koestler, engagement in virtual museums

24. Conclusions In virtual reality striving to achieve More realism While maintaining real-time performance Many applications don’t need physical realism For those that do May not be possible to compute total physical realism Perceptual realism Same user response as if there in real scene being portrayed Can save significant computation by exploiting human perception

25. Challenges Delivery system Virtual cocoon (light weight, portable system) Determine the “weights” ωiin the perception equation Empirical studies fMRI brain imaging Weights are individual Some generic similarity Do NOT have to be determined precisely As long as we are ABOVE the threshold Perceptually high-fidelity virtual environments Real virtuality

26. Acknowledgements Pavel Zemcik and MEMICS team for inviting SCCG for enabling long-term discussions EPSRC for funding “…towards Real Virtuality” Alan’s group for 23 years of exciting research David Howard + Christopher Moir +red wineReal Virtuality Further details: alan.chalmers@warwick.ac.uk ferko@sccg.sk

27. Levels of Realism: From Virtual Reality to Real Virtuality -Perception and Virtual Time Alan Chalmers† Andrej Ferko‡ † Warwick Digital Laboratory, UK ‡ Comenius University, Slovakia