Tae Soo Yun Dept. of Digital Contents Dongseo University Fall 2002 based on notes from

1. Advanced Topics in Virtual Reality Tae Soo Yun Dept. of Digital Contents Dongseo University Fall 2002 based on notes from Soon Ki Jung, KNU Wohn, KAIST ……

2. Table of Contents • Introduction : What is VR ? • Psychological and Cognitive Issues • VR System Anatomy • Virtual Perception • Interaction • Virtual Worlds: Representation, Creation and Simulation • Virtual Worlds: Rendering • Networked VR Systems and Shared Virtual Environements • Image-based Virtual Reality • Augmented Reality

3. Conceptual Model of VR Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor V-effector sensing action P-effector L-effector sensing output device input device action P-sensor L-sensor joystick mouse tracker TV camera microphone 2D - selector pick locator etc. 3D - navigator manipulator etc. V-human V-vehicle V-bike V-hand etc.

4. Basic model of perception & cognition sensing H-sensor perception cognition motion control H-effector knowledge action Human Natural environment

5. 2-1. Human Visual System 2-2. Human Auditory System 2-3. Other Perceptual Systems 2-3-1. Tactile 2-3-2. Vestibular 2-3-3. Kinesthetic 2-3-4. Olfactory 2-3-5. Gustatory 2-4. Human Motor System 2-5. Cognitive System 2-6. Presence and Reality

6. 2-1. Human Visual System 1. Outline of the Visual System 2. Eye 3. Properties of Eye 4. Properties of Visual System 5. Spatial Perception

7. 1. Outline of the Visual System • Properties • Bandwidth • Sensitivity • Visual Path • Eye • Retina • Optic nerve • Lateral Geniculate Nucleus • Visual cortex

8. 2. Eye • Cornea (각막) • Iris (조리개) – pupil opening 제어 • Pupil opening (동공) • 지름2-8 mm, 통과하는 빛의 양 결정 • 렌즈의 지름이 변하므로 depth-of field가 변함 • Lens (수정체) • 렌즈의 두께를 변화시킴으로써focal length변화 • 비교적 느림 (400 msec from nearvision to far vision) • Retina (망막) • 약 10 neural layers

9. receptor

10. 3. Properties of Eye System • Accomodation and depth of field • Thresholding intensity • Spectral sensitivity • Dynamic range • Dark adaptation • Visual field • Visual acuity • Spectral response • Eye movements

11. Properties of Eye System (cont.) • accomodation • adjust the focus by changing the lens curvature. • Eye measure the amount of “blur”. • Accomodation does not occur in the graphic scene. • Eye의 근육이 이완되어 있는 상태에서는 depth of field가 6 m ~ 무한대이므로 다른 노력 없이 in-focus 가능. • 동공의 지름 = 8 mm: depth of field = 6m – 무한대 • 동공의 지름 = 4 mm: depth of field = 3.5m – 무한대 • 동공의 지름 = 2 mm: depth of field = 2.3m - 무한대 • 물체가 6m이내에 들어오면 focus를 맞추기 위해 근육을 수축하여 렌즈의 곡률을 변화시킴. • 물체까지의 거리가 1m 일 때: • 동공의 지름 = 4 mm: depth of field = 0.8m - 1.4m • 동공의 지름 = 2 mm: depth of field = 0.7m - 1.8m

12. Properties of Eye System (cont.) • dynamic range • 인간이 sense하는 intensity의 dynamic range = 1:10^13 • 그러나 한 순간에서의 operating range = 1: 100 • Discriminating power = 1% • dark adaptation • 어두운 환경에 갑자기 들어 갈 때 감지 할 수 있는 빛의 최소량은 시간에 따라 감소. • Cone -- 처음 7-8 분간 빨리 반응. • Rod -- Adaptation은 느리나 더 적은 빛에서도 반응. Cone보다 약 100배 sensitive함. • 따라서, 어두운 곳에서는 • Color perception이 잘 안됨 • 붉은 색보다는 푸른색이 눈에 잘 띔 • Resolution이 떨어짐

13. Properties of Eye System (cont.) • visual field • visual angle • visual field • one eye: 150 degrees horizontal, 120 degrees vertical • two eyes: 200 degrees 120 • overlapping field: +/- 30 ~ 40 degrees

14. Properties of Eye System (cont.) n r s d Nodal point Visual Angle

15. Properties of Eye System (cont.) Visual field of a normal human observer.

16. visual acuity • ability to identify / discriminate features. • 20/20 vision = -정상인이 20 ft 떨어진 곳에서 식별할 수 있는 정도를 피실험자도 20 ft거리에서 식별. • 20/100 vision = 정상인의 100ft에서의 식별 능력과 피실험자의 20ft에서의 식별 능력이 같음. • recognition acuity • Feature를 식별할 수 있는 능력 • 최대 1' (100/100 vision) 2) detection acuity • Single line을 detect할 수 있는 능력 • 최대 2' 3) resolution acuity • Double lines을 detect할 수 있는 능력 • 최대 30’’

17. spatial resolution • obtained from visual acuity • 4,800 x 3,800 • Spatial frequency response • Sine grating의 frequency를 바꿔가며 인간이 감지할 수 있는 최소의 contrast (즉, sine grating의 amplitude)를 측정한다. • 최대의 sensitivity는 4 - 5 Hz에서 발생.

18. Eye Movements 1) Nystagmus (안진) • Very fast (30 - 100 Hz) , Very small (약 17") 2) Saccades • Drift를 보정하는 운동, 또한 target이 움직일 때 추적하는 운동. • 초당 3 - 5 회 발생.1초에 100msec은 saccade운동, 900msec은 정지. • Movement중에는 image capture하지 않음. • Very fast (40° in 100 msec) 3) Drift • Fixation point가 target 으로부터 조금씩 벗어나는 현상. • Slow (3'/sec) 4) Pursuit move • 자발적인 운동으로서 물체가 움직일 때 계속 추적하는 운동. • Eyeball의 움직임 range • Yaw: ±50° , Pitch: 40° above, 60° below. , Roll: 6°

19. 4. Properties of Visual System • Feature detection • Edge detection • Corner detection • Motion • Short-term motion -- Detection • up to 10’ –15’visual arc,up to 80 – 100 msec interval • Long-term motion -- Matching • Apparent motion = perception of motion without stimulus continuity • Color perception • Physical stimuli vs. Color perception

20. Non-pictorial depth cues Accomodation 유효거리: 2-3m Stereopsis convergence disparity Motion parallax Pictorial depth cues size consistency perspective arial perspective luminance shading light and shadow color texture occlusion 5. Spatial Perception

21. Stereopsis (binocular vision) = the process that “integrate two images into one perception” • binocular rivalry • Visual system suppresses one image when two images are very different • Caused by the luminance difference and others • Human has the dominant eye • binocular convergence • 약 6m – 9m 까지 유효 • binocular disparity • Disparity가 1° 이하일 때는 상당히 정확한 식별 가능. • 50 cm 거리에서는 0.05 mm 분별,5 m 거리에서는 4 mm 분별 • Disparity가 1° 이상일 때는 상대적인 거리만 판별. • Disparity가 5° 이상이 되면 matching 불능

22. Motion • motion parallax • 약 50m까지 유효 • depth from motion • Kinetic depth effect • Stereo-kinetic depth illusion

23. Fixation point Right Eye Left Eye Disparate images of far rod Foveas Fixated rod Disparate rod

24. 2-2. Human Auditory System • Characteristics of Sound Sound wave = the change of atmospheric pressure Atmospheric pressure = 10^6  bar Detectable range of ear = 0.0002  bar - 200  bar 2. Anatomy of Auditory System • outer, middle, and inner ears. 3. Psycho-acoustics: Auditory Perception • physical -- sensitivity, discriminability • psychophysical -- loudness, pitch, timbre • precedence effect, cocktail party effect • Spatial Hearing

26. Sound localization • azimuth, elevation, distance • Azimuth perception is due to … • Time difference • Intensity difference • Monaural cues derived from the pinna(귓바퀴) • Head/sound movements • Interaural time difference • Max time difference = 660 msec(time for reaching from one ear to the other) • Min discriminating time = 10  sec • Effective for low frequency range

27. Sound localization (cont.) • Interaural intensity difference • Max intensity difference = 20 dB (due to head occlusion) • Min discernable intensity difference = 0.7 dB • Effective for high frequency range • Cues derived from the outer ear (especially pinna(귓바퀴)). • HRTF (Head related transfer function) • Distance perception is due to … • Intensity cues • Spectral changes • Auditory parallax

28. 2-3. Other Perceptual Systems 1. Tactile • sense of touch, pain, cold, warmth. • Five different skin receptors are responsible for the basic senses 2. Kinesthetic • movements and relative positions of body parts. 3. Vestibular • sense of acceleration, orientation, rotational motion. 4. Olfactory • sense of smell. 5. Gustatory • sense of taste.

29. Tactile Perception • Sense of touch • Basic sensing unit = pressure • Five different skin receptors are responsible for the basic senses: • Touch • Pain • Cold • Warmth • Usage • How to design the tactile display system ?

31. Kinesthetic Perception • Kinesthesia = awareness of the movements and relative position of the various body parts • Internal sensing of kinematic (position, angle) and dynamic (speed, force, weight) status • Due to the sensory receptors in joints, muscle and skin • Joints closer to body (proximal joints) are more sensitive than distal joints (such as fingers) • Usage • How to design the force feedback system? • How to present a virtual hand ? • Visual perception + kinesthetic perception

32. Vestibular (equilibratory) perception • Vestibular sense = sense of balance and position • Acceleration • Orientation • Rotational motion • Usage • How to design the motion platform ?

33. Olfactory • Smell classification ? • Not yet!

34. Gustatory • Basic taste qualities • Salty, sour, sweet, bitter • Hey, are any of these related to computer science, even remotely?

35. 2-4. Human Motor Systems Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor V-effector sensing action P-effector L-effector sensing output device input device action P-sensor L-sensor joystick mouse tracker TV camera microphone V-human V-vehicle V-bike V-hand etc.

36. Theoretical basis for planning, modifying and executing motor movements. 1. Reaction time 2. Motor control • movement time as the time for processing information • Fitts’ law: movement time = a + b log(2·D/W) 3. Kinematic properties 4. Dynamic properties

37. Reaction time • Donder’s experiments • Reaction A • One stimulus -> one response • Reaction B • Multiple stimuli -> multiple responses (e.g., at a traffic light) • Reaction C • Multiple stimuli -> single response (e.g., name calling in the class) • Reaction time = nervous system conduction time + time for mental identification + time for mental selection

38. Reaction time (cont.) • Information theoretical approach • Reaction time information amount (Hick’s law) • Reaction time # of alternatives • Determinants of reaction time • Stimulus-response compatibility • Practice • Speed-accuracy trade-off

39. Motor control • Fitts’ law • Movement time = a + b log(2D/W) • … is an application of information theory to the problem of motor control (Motion generates information) • Motor program • Program that generates motion by accepting the control parameters (e.g., pianist, athletes)

40. Kinematic properties • Eye movement • Head movement • 600°/sec for yaw, 300°/sec for pitch and roll • Body movement • Locomotion

41. Dynamic properties • Force • Torque

42. 2-5. Cognitive System Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor V-effector sensing action P-effector L-effector sensing output device input device action P-sensor L-sensor joystick mouse tracker TV camera microphone V-human V-vehicle V-bike V-hand etc.

43. Why do we care about? • Designing interface – human can transmit information only at a finite rate. • Developing the virtual perception. • We are interested in … • Capacity • Structure • Disciplines on human cognition • Philosophy – thinking as a “logical” system • Psychology – biologically (or neuro-physiologically)-based model • Human factors – cognition as information processing • AI – cognition as symbolic manipulation

44. Memory • Sensory memory • Holds information from eyes and ears • Iconic • Short-term memory (working memory) • Limited capacity (5-9 items) • Long-term memory • Permanent storage • No guarantee for accessibility

45. Attention • Selective attention • Ability to focus (or ignore) some events • Divided attention • Ability to do more than one thing at the time

46. Mental workload • Mental workload vs. physical workload • Measuring the mental workload • Subjective opinion: rating scale, questionnaires • Spare mental capacity: based on the assumption that human channels have the upper bound • Primary-task method: based on the assumption that increasing mental load will cause a decrement in primary-task performance

47. 2-6. Presence and Reality • VR: the medium • Tele-presence and Virtual presence • AIP cube • Model P • Measuring reality • Philosophical considerations

48. 1. VR: The medium • The word medium means to be between two ends/extremes. • In the case of human communication, it is the process by which ideas are manifested and then experienced by people.

49. The content of HC medium is.. • Examples of virtual worlds implemented in various media: • Imagination : Day dream / Mental model • Imagination/Toys : Child playing with a doll • Storytelling • Cave-painting • Novel (non-interactive fiction) • Maps • Illusions (of magic) : Pulling rabbit out of a hat • Song • Motion Picture (non interactive) • Animation • Puppetry • Ham Radio • Interactive-Fiction (via book) • Interactive-Fiction (via computer) • E-mail • Newsgroups (via internet) • Multimedia Interactive-Fiction …….

50. Communicating ideas through a medium