1 / 76

Tae Soo Yun Dept. of Digital Contents Dongseo University Fall 2002 based on notes from

Advanced Topics in Virtual Reality. Tae Soo Yun Dept. of Digital Contents Dongseo University Fall 2002 based on notes from Soon Ki Jung, KNU Wohn, KAIST ……. Table of Contents. Introduction : What is VR ? Psychological and Cognitive Issues VR System Anatomy Virtual Perception

dana-mccall
Download Presentation

Tae Soo Yun Dept. of Digital Contents Dongseo University Fall 2002 based on notes from

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Advanced Topics in Virtual Reality Tae Soo Yun Dept. of Digital Contents Dongseo University Fall 2002 based on notes from Soon Ki Jung, KNU Wohn, KAIST ……

  2. Table of Contents • Introduction : What is VR ? • Psychological and Cognitive Issues • VR System Anatomy • Virtual Perception • Interaction • Virtual Worlds: Representation, Creation and Simulation • Virtual Worlds: Rendering • Networked VR Systems and Shared Virtual Environements • Image-based Virtual Reality • Augmented Reality

  3. Functional diagram rendering (chap. 7) displaying (Sec. 3-4,5,6,7) simulation (Sec. 6-3,4) VW DB (Sec. 6-1) Virtual perception (chap. 4) interaction (chap. 5) Sensing (Sec. 3-3) VW Authoring (Sec. 6-2)

  4. sensing and virtual perception Glove Controller Hand gesture recognition Video camera Image processor Body gesture recognition Video camera Image processor Facial/face expression recognition Microphone Signal processor Speech recognition Sensing Virtual perception

  5. 4-1. Hand Gesture Recognition 4-2. Body Posture and Gesture Recognition 4-3. Face and Facial Expression Recognition 4-4. Gaze Tracking 4-5. Speech Recognition

  6. 4-1. Hand Gesture Recognition 1. Classification of Hand Movements 2. Issues on Gestural Communication 3. Recognition Technology

  7. Examples • Pointing to real and abstract objects and concepts • Waving, saluting, praying (two flat hands up together) • Live or die decisions in the Roman amphitheater (thumb up/down) • Counting (fingers and/or hand) • Rejective (index up moving left and right) • Appreciative (hand clapping) gestures • Traffic control signs • Conducting of an orchestra • sign languages

  8. 1. Classification of Hand Movements • Semiotic Hand gesture • to communicate meaningful information and results from shared cultural experience • Ergotic (Manipulative) • Epstemic • tactile experience or haptic exploration • Hand gestures • Movements that we make with our hands to express emotion or information, either instead of speaking or while we are speaking. ==>

  9. Hand Gestures • Kendon (1988) : in regard to the levels of linguisticity • spontaneous gesture (gesticulation) • language-like gestures • pantomimes • emblems • sign languages • McNeill (1992) : further classification of gesticulation • iconics • metaphorics • deictics • beats

  10. Nespoulos and Lecours (1986) • arbitrariness • Arbitrary gestures : need to be learned • Mimetic gestures : common within a culture • Deictic gestures (specific / generic / function indication) • function • Quasi-linguistic • Coverbal expression - illustrative / expressive / paraverbal • Social interaction - phatic / regulatory • Meta-communication • Extra-communication

  11. Hand Motion intentional movement unintentional movement direct manipulation communicative (hand gesture) [Kendon(1988)] spontaneous (natural) emblems language-like gesture sign language pantomimes iconic metaphoric deictics beats [McNeill(1991)] spatiographic kinetographic pictographic Summary of Classification

  12. Pictographic gestures Deictics rectangular shape round shape triangle shape Kinetographic gestures Metaphoric gestures rotate delete or cancel come-closer move-up down move-right move-left move-far fixation Example Gestures

  13. 2. Issues on Gestural Communication • Context detection • Identifying hand gestural attributes • Tracking of hand gesture • Low-level (visual pattern) interpretations • Integration of the low-level interpretations • Gesture segmentation • Correlating the hand motion and virtual world • Integration (or fusion) with other modalities

  14. (1) Context detection • Global context detection • When to look at the user’s hand movement? • Existence of speech(Gesticulation 의 경우) • Use speech semantic functions • (eg.) “Move the ball from the table to there” • “Move the brush like this” • Hands’ movement into gesture region

  15. Context within gesture • Gestural phases [Kendon 1980] [MIT1996] - preparation - pre-stroke hold - stroke - post-stroke hold - retraction

  16. (2) Attributes of hand gestures • Static attributes • Hand configuration (posture) • Hand orientation • Hand position relative to the body • Non-static attributes • Movement direction • Movement speed / acceleration • Magnitude of movement • Shape (arm movement trajectory) • linear / nonlinear

  17. Pattern characteristicsof hand gesture • Multiple concurrent attributes • Spatio-temporal dynamicity • Spatial property - 3D Shape variation - 3D Orientational variance - 3D Positional variance • Temporal property - Temporal variance : Interperson / Intraperson

  18. (3) Tracking of hand gesture • Sensor-based technology • Gloves • Position Tracker • Vision-based technology y y 10 flex angles x x z z

  19. (4) Low-level interpretation • Classical pattern recognition problem, or not? • Similar to the low-level vision.

  20. (5) Integration of low-level interpretations

  21. (6) Segmentation • Similar to the intermediate-level vision problem.

  22. (7) Context-sensitive interpretation • A gesture may have different meanings in different contexts.

  23. (8) Multi-modal integration • Integration of hand gesture, gaze, speech, body movement.

  24. 3. Recognition technology • Isolated • continuous • action-completion based interpretation • action-following (e.g., conducting) • one-hand • two-hand

  25. Static attributes (posture and orientation) • Feature-based approach [Roy94] • statistical approach [Newby94][Darrell95] • Neural network [Vaananen93][Kim95]

  26. Non-static attributes • Feature-based approach [Rubine91] • Neural network [Murakami91][Waldron95] • Statistical approach [Wilson95][Starner95]

  27. Gesture • posture transition • Statistical correlation • Neural network-based recognition at every moment, and then temporal integration

  28. 1/4 1/2 transition probability 1 1/2 1/2 1/4 0.4 0.2 0.3 0.3 0.3 0.4 0.5 0.4 0.1 a b c observation probabilities • Gesture • arm movement • Fuzzy configuration states [Wilson95] • Hidden Markov Models [Starner95][Nam96][Wilson96]

  29. Posture+Movement at the same time • Recurrent neural network [Taguchi91] • Two-level back-propagation network [Waldron95]

  30. * Difficulties Isolated (single stroke) Linear movements only Increasing recognition difficulty in engineering point of view (n ³ 1 strokes) Rotary or shaping movement connected (multiple strokes) Repetition of gesture Continuous gesture Temporal dynamicity (with significant velocity, acceleration)

  31. Continuous arm movement • HMM Network [Nam96] .... Global Final Node Global Initial Node ConnectingHMMs Null Transitions Null Transitions .... pause pause

  32. Temporal integration for continuous recognition • [Wexelblat 95]

  33. Recognition & Integration Situation-Tracker high-level meaning in application token giving environment perception by finding current marking of GPN Gesture-level Tracker whole gesture pattern [GUARD] [GUARD] AttributeTracker Posture Classifier each attribute pattern HMM network Orientation Classifier

  34. Further issues • Automatic segmentation of fully continuous gesture • Two-hand gesture • Multi-modal interaction

  35. 4-2. Body gesture recognition • TPs and lecture notes of Section 4-2 have been prepared by Chang-Whan Sul. • human body as input devices in VR • physical control over the avatar • task / command level interface

  36. approaches (tracking) • electromagnetic / acoustic / mechanical trackers • simple, tracks accurately in realtime • invasive, restricted area, distorted signal • vision-based human body tracking • non-invasive, virtually no limit in working area • computationally heavy • non-realtime when precision is required • rough recognition when realtime performance required

  37. research on body gesture • Psychology • MLD(Moving Light Display) interpretation • 3D structure from motion • gait, gender • person identification • Computer Vision • Restricted class of motions • Gait analysis • Sports (Tennis action) • Simple general motions • HIA(Humans In Action) • IVE(Interactive Video Environment) • MandalaTM

  38. hurdles in vision-based body gesture • human body motion • articulated motion • too many DoF’s • non-linear, non-rigid motion • Limitation of vision technology • well controlled environment needed. • ex - uniform/stationary background, controlled illumination

  39. classification of existing works

  40. Procedures Feature extraction tracking Feature matching (inter-frame/inter-view) 3-D model reconstruction recognition

  41. Methods • Template matching • Feature vector classification • Connectionist approach • Neural Network • Probabilistic approach • Hidden Markov Model • Proper use of domain-specific knowledge on the human body is the key.

  42. pelvis Tw,Rw L_thigh R_thigh chest Rp L_calf R_calf L_u_arm R_u_arm head L_forearm R_forearm Motion Analysis of Articulated Objects forOptical Motion CaptureSoonKi Jung, VR Group, KAIST 1997 • model-based, 3D model, point tracking 3-D human model

  43. Track point features from multiple cameras by HIEKF(hierarchical iterative extended Kalman filtering)

  44. Captured motion used to control avatars • motion transition / blending / variation issues • adjust to various avatar model

  45. Spfinder (stereo person finder)Ali Azarbayejani, C.Wren, A Pentland, MIT Media Lab. • used in ALIVE and the descendants

  46. 2-D blob tracking(pfinder) • 2-D blob with same color • domain knowledge about human body • tracks head, hands, feet • Kalman Filter

  47. 3-D blob tracking • based on 2-D blob tracking • Self-calibrates 3-D blob correspondences

  48. HIA (Humans In Action)D.M. Gavrila & L.S. Davis (Univ. Maryland) • 3-D model-based • skeleton + tapered superquadric • specially designed suite • multiview generate-and-test strategy • similarity measure: chamfer distance • edge-based feature • hierarchical search sampled space around current model • used to build motion DB

  49. model projection view model view model edge edge edge edge 1 1 n n similarity similarity 1 n similarity Movie file URL ftp://vr.kaist.ac.kr/pub/cs778/{d_circlwalk,d_twoelbrot, e_twoelbrot}.mpeg

  50. First SightM.K. Leung & Y.H. Yang • 2-D model • stick figure + ribbon-represented body • ribbon using moving edge • body labeling constraints • structural I - joint deviation • structural II - combination • shape constraints • balance

More Related