Visual Perception in Realistic Image Synthesis

1. Visual Perception in Realistic Image Synthesis Ann McNamara

2. Outline • Introduction • Modeling important characteristics of the human visual system (HVS) • Perception based rendering • Image quality metrics • Tone reproduction operators • Summary

3. Realism • Architecture • Stage lighting • Entertainment • Safety systems • Archaeology

4. Human Visual System • Physical structure well established • Perceptual behaviour is a complex process

5. Modeling Important Characteristics of the Human Visual System

6. Visual Acuity • How well we can see fine detail • Adaptation level • Rods and cones Cones

7. Spatial Frequency • Number of grating that fall on one degree of the retina • Dependent on distance

8. Spatial Frequency • Spatial mechanisms (channels) which are used to represent the visual information at various scales and orientations as it is believed that primary visual cortex does.

9. Contrast Sensitivity Function • Contrast sensitivity function which specifies the detection threshold for a stimulus as a function of its spatial frequencies.

10. Contrast Sensitivity Campbell-Robson contrast sensitivity chart

11. Masking • Visual masking affecting the detection threshold of a stimulus as a function of the interfering background stimulus which is closely coupled in space and time.

12. Masking

13. Colour Appearance

14. Perceptually Based Rendering

15. Low Sampling Densities [Mitchell 1987] • Non-uniform sampling is less conspicuous • Optimise using how the eye perceives noise as a function of contrastand colour Raytracing -> Point Samples-> Aliasing

16. Uniform Non-Uniform Adaptive Sampling Schemes [Mitchell 1987]

17. Low Sampling Densities [Mitchell 1987] • Contrast • Colour R 0.4G 0.3B 0.6

18. Low Sampling Densities [Mitchell 1987]

19. Frequency Based Raytracing [Bolin &Meyer 1992] • Synthesise directly into frequency domain • Simple vision model to control • Where to cast rays • How to spawn rays

20. Frequency Based Raytracing [Bolin &Meyer 1992] • Vision model • Contrast sensitivity • Spatial frequency • Masking

21. Frequency Based Raytracing [Bolin &Meyer 1992] • Specific luminance difference at low intensity more important than same luminance difference at high intensity • Colour spatial frequency variations given fewer samples • Decrease rays spawned in high frequency regions

22. Colour Abberation Limited sampling of receptor Spatial acuity of opponent channels Limited Color Acuity [Meyer & Liu1998]

23. [Meyer & Liu1998]

24. Application • How much computation is enough? • How much reduction is too much? • An objective metric of image quality which takes into account basic characteristics of the HVS could help to answer these questionswithout human assistance.

25. Questions of Appearance PreservationThe Concern Is Not Whether Images Are the Same Rather the Concern Is Whether Images Appear the Same

26. Perceptually Based Adaptive Sampling Algorithm [Bolin &Meyer 1998] • Image quality model embedded into image synthesis • Use statistical information about spatial frequency to determine where to estimate values where samples were yet to be taken

27. VDM JND’s Perceptually Based Adaptive Sampling Algorithm [Bolin &Meyer 1998]

28. Convergence Evaluation [Myszkowski 1997] Deterministic radiosity  = 200s  = 400s  = 800s  = 1600s Monte Carlo radiosity

29. Termination Criterion [Myszkowski 1997] 0.5 vs.   vs. reference

30. Physical Based Perceptual Metric [Ramasubramanian et al1999] • Threshold model defines a physical error metric • Handles luminance-dependent and spatially dependent processing independently • Allowing pre-computation of spatially-dependent component

31. Physical Based Perceptual Metric [Ramasubramanian et al1999]

32. Image Quality Metrics

33. Image Quality • Compare and validate lighting simulations • Use comparisons to guide rendering more efficiently • Compute less without altering perception • Pixel by pixel comparison might be > 0, human might not see any difference

34. RMSE 9.5 RMSE 5.2 Pixel by Pixel Comparison Prikryl, 1999

35. Amplitude Nonlinear. Contrast Sensitivity Function Cortex Transform Masking Function Unidirectional or Mutual Masking + Psychometric Function Probability Summation Visualisation of Differences Amplitude Nonlinear. Contrast Sensitivity Function Cortex Transform Masking Function Visible Differences PredictorVDP [Daly ‘93, Myszkowski ‘98] Image 1 Image 2

36. Pixel differences: Standard - Comparison Comparison Pixel differences The VDP response: probability of perceiving the differences Standard VDP response VDP: Results

37. Daly’s VDP: Features Daly, 1993 • Predicts local differences between images • Takes into account important visual characteristics: • Amplitude compression • Advanced CSF model • Masking • Uses the cortex transform, which is a pyramid-style, invertible & computationally efficient image representation

38. Visible Discrimination Model Lubin, 1997 • Map of Just Noticeable Differences • Point sample function to model optics • Resample the image according to foveal eccentricity • Band pass response • Contrast pyramid • steerable filters

39. Visible Discrimination Model Lubin, 1997 • Both images subjected to Identical processing • Distance measure • Difference in responses for Each channel and summing Them to obtain a JND Map of the two images

40. An Experimental Evaluation of Computer Graphics Imagery [Meyer et al, 1986] • Comparing image to real-world scene • An approach to image synthesis consisting of • A physical module • A perceptual module

41. Simulated Measured Difference An Experimental Evaluation of Computer Graphics Imagery [Meyer et al, 1986]

42. An Experimental Evaluation of Computer Graphics Imagery [Meyer et al, 1986]

43. An Experimental Evaluation of Computer Graphics Imagery [Meyer et al, 1986]

44. Image Quality Metrics [Rushmeier et al, 1995] • Components of perceptually based metrics adapted from image compression • Gervais et al 1984 • Mannos et al 1974 • Daly 1993

45. Real Room Simulated Model of Room Image Quality Metrics [Rushmeier et al, 1995] • Daly tested very well

46. Visual Psychophysics • Determine the relationship between the physical world and human’s subjective experience of that world • Measure the response (“psycho”) to a known stimulus (“physics”)

47. Why Lightness ? [Gilchrist 1977]

48. A Psychophysical Investigation [McNamara et al 1998, 2000] • Painted 5-sided cube • Objects painted with different grey paints • Complex illumination, with secondary reflections

49. Graphic Reconstructions [McNamara et al 1998, 2000]

50. Real Scene Rendered Experiment [McNamara et al 1998, 2000]