610 likes | 740 Views
Contribution to the study of visual, auditory and haptic rendering of information of contact in virtual environments. 9/12/2008. Jean Sreng. Advisors: Claude Andriot Anatole Lécuyer. Director: Bruno Arnaldi. Introduction. Context: Manipulation of solid objects in Virtual Reality
E N D
Contribution to the study of visual, auditory and haptic rendering of information of contact in virtual environments 9/12/2008 Jean Sreng Advisors: Claude Andriot Anatole Lécuyer Director: Bruno Arnaldi
Introduction • Context: Manipulation of solid objects in Virtual Reality • Example applications: industrial virtual assembly / disassembly / maintenance • Focus: perception, simulation, rendering and of contacts between virtual objects
Outline • State of the art on perception, simulation and rendering of contact • Contributions • Integrated 6DOF multimodal rendering approach • Visual of rendering of multiple contacts • Spatialized haptic rendering of contact • Conclusion Integrated 6DOF Multimodal rendering OUTLINE Visual rendering of multiple contacts Spatialized haptic rendering
Human perception of contact Perception of contact Simulation of contact Rendering of contact • Visual perception of contact: • Stereoscopy (Hu et al. 2000) • Motion parallax (Wanger et al. 1992) • Shadows (Wanger et al. 1992, Hu et al. 2002) • Auditory perception of contact: • Contact properties can be directly perceived (Gaver 1993) • Contact sounds conveys information about shape and material (Klatzky 2000, Rochesso 2001)
Haptic perception Perception of contact Simulation of contact Rendering of contact • Haptic perception provides an intuitive way to feel the contact(Loomis et al. 1993) : • Tactile perception (patterns at the surface of the skin) • Kinesthetic perception (position and forces) • Physical properties can be perceived through the contact (Klatzky et al. 2003): • Shape / Texture / Temperature • Weight / Contact forces • Perception of contact features through vibrations • Material (Okamura et al. 1998) • Texture (Lederman et al. 2001)
Multimodal perception of contact Perception of contact Simulation of contact Rendering of contact • Known Interaction between modalities • Visual-Auditory interaction • Ex: Sound can shift the perception of impact (Sekuler et al. 1997) • Visual-Haptic interaction • Ex: Pseudo-haptic feedback (Lécuyer et al. 2002) • Auditory-Haptic interaction • Ex: Sound can modulate the perception of roughness (Peeva et al. 1997) “Stiff” “Soft”
Simulation of contact Perception of contact Simulation of contact Rendering of contact • Multiple contact models (impact/friction) • Rigid (Newton/Huygens impact law, Coulomb/Amontons friction law) • Locally deformable (Hunt-Crossley impact law, LuGre friction law) • Multiple simulation methods • Collision detection • VPS(McNeely 1999) • LMD (Johnson 2003) • Physical simulation • Constraint based (Baraff 1989) • Penality (Moore 1988)
Visual rendering Perception of contact Simulation of contact Rendering of contact • Visual rendering of the information of contact • Proximity : • Color (McNeely et al.2006) • Contact : • Color (Kitamura et al.1998) • Glyph (Redon et al. 2002) • Force : • Glyph (Lécuyer et al. 2002)
Auditory rendering Perception of contact Simulation of contact Rendering of contact • Realistic rendering of contact sounds • Specific: Impact/Friction/Rolling • Different techniques: FEM(O’Brien 2003), modal synthesis(Van den Doel 2001) • Symbolic rendering (Richard et al. 1994, Massimino 1995, Lécuyer et al. 2002) • Associate an information to a sound effect • Information: Distances / Contact / Forces • Sound effect: Amplitude / Frequencies
Haptic display of contact Perception of contact Simulation of contact Rendering of contact • Haptic devices (Burdea 96) : • Force feedback • Tactile feedback • Haptic rendering of contact : • Closed loop (McNeely et al. 1999, Johnson 2003) tradeoff between stability / stiffness • Open loop (Kuchenbecker et al. 2006) improve the realism of impact Virtual Env. Device Impact
Objectives of this thesis • Improve the simulation, rendering, and perception of contacts in virtual environments Integrated 6DOF multimodal rendering • Protocol : • Integrated 6DOF approach for multisensory rendering techniques • Study of rendering techniques to improve the perception of contact position • Hypothesis of improvement • Experimental implementation • Experimental evaluation Visual rendering of multiple contacts Spatialized haptic rendering
Outline Integrated 6DOF multimodal rendering OUTLINE Visual rendering of multiple contacts Spatialized haptic rendering
Objectives • Multiplicity of techniques: • Contact simulation • Sensory rendering • How can we integrate all techniques seamlessly together independently from the simulation ? • Contribution / Overview • Contact formulation (states / events) • Example of contact rendering based on this formulation : Visual, Auditory, Tactile, Force feedback Integrated 6DOF multimodal rendering
Contact formulation • Simple contact formulation based on : • proximity points a, b • Force f Integrated 6DOF multimodal rendering
Contact formulation • From this formulation : • Contact states • Temporal evolution of contact states (such as events) : • Higher level information • Adapted to many specific rendering techniques Integrated 6DOF multimodal rendering Free motion Contact Impact Detachment
Determination of states and events Contact Impact Detachment Integrated 6DOF multimodal rendering • The contact condition : • The events are defined by : • Impact : • Detachment : Local linear velocity Normal
Multimodal rendering architecture Integrated 6DOF multimodal rendering
Multimodal rendering architecture • Superimpose states and events information: Integrated 6DOF multimodal rendering
Example of Visual rendering Contact Impact Detachment Integrated 6DOF multimodal rendering
Example of Auditory rendering Integrated 6DOF multimodal rendering
Multimodal rendering platform • Multimodal: • Visual • Auditory • Tactile • force-feedback Integrated 6DOF multimodal rendering © Hubert Raguet / CNRS photothèque
Preliminary conclusion • We proposed a contact formulation (proximity/force) : • Contact states and events • We developed a multimodal rendering architecture : • Visual (particles / pen) • Auditory (modal synthesis / spatialized) • Tactile (modal synthesis) • 6DOF Haptic enhancement (openloop) Integrated 6DOF multimodal rendering
Outline Integrated 6DOF Multimodal rendering OUTLINE Visual rendering of multiple contacts Spatialized haptic rendering
Objectives • Context: complex-shaped objects • Multiple contacts • Difficult interaction • Help the user by providing position information • Contribution / Overview • Display the information of proximity / contact / forces • Glyphs • Lights • Subjective evaluation Visual rendering of multiple contacts
Visual rendering of multiple contacts • Visualizing multiple proximity / contact / forces positions Proximity Contact Contact forces FAR NEAR LOW HIGH Visual rendering of multiple contacts
Visual rendering using glyphs Proximity Contact Contact forces FAR NEAR LOW HIGH Visual rendering of multiple contacts
Visual rendering using glyphs Visual rendering of multiple contacts
Glyph filtering • Reduce the number of displayed glyphs • determine “relevance” based on the movement Visual rendering of multiple contacts
Glyph filtering • Reduce the number of displayed glyphs • determine “relevance” based on the movement Visual rendering of multiple contacts
Glyph filtering • Reduce the number of displayed glyphs • determine “relevance” based on the movement Visual rendering of multiple contacts
Glyph filtering • The relevance is determined by comparing : • The local velocity v • The local normal d d v Visual rendering of multiple contacts
Visual rendering using lights • Two types of lights : • Spherical lights • Conical lights Visual rendering of multiple contacts
Visual rendering using lights Visual rendering of multiple contacts
Subjective evaluation • Objective: Determine user’s preferences about the different techniques • Procedure: Participants were asked to perform an industrial assembly operation • Without visual cues • With each visual cues • Conducted on 18 subjects • They had to fill a subjective questionnaire • Which effect : glyph / light / color change / size change / deformation • For which info : forces / distances / blocking / focus attention Visual rendering of multiple contacts
Results • The visual effects were globally well appreciated • Significant effects : • Glyph Size effect globally appreciated (distance / force) • Glyph Deformation effect appreciated to provide force information • Light effect appreciated to attract visual attention Mean ranking “Lower is better” Visual rendering of multiple contacts
Preliminary conclusion • We proposed a visual rendering technique to display multiple contact information • Display Proximity / Contact / Force • Using Glyphs/Lights • We presented a filtering technique to reduce the number of glyphs displayed • We conducted a subjective evaluation • Glyph size: proximity / force • Glyph deformation: force • Lights: focus the visual attention Visual rendering of multiple contacts
Outline Integrated 6DOF Multimodal rendering OUTLINE Visual rendering of multiple contacts Spatialized haptic rendering
Objectives • Context: Complex-shaped objects • Help the user by providing position information • Provide contact position information using : • Visual rendering (particles/glyphs/lights) • Auditory rendering (spatialized sound) • Can we provide this position information using haptic rendering ? • Contribution / Overview • Haptic rendering technique based on vibrations • Perceptive evaluation in a 1DOF case • 6DOF Haptic rendering technique • Perceptive evaluation to determine rendering parameters • Subjective evaluation in a 6DOF case Spatialized haptic rendering
Haptic rendering of contact position • The impact between objects : • A reaction force of contact • A high frequency transient vibrations • This high-frequency transient vibrations depends on : • The object’s material (Okamura et. al 1998) • The object’s geometry • The impact position • Is-it possible to perceive the impact position information using these vibrations? Spatialized haptic rendering
Vibrations depending on impact positions • Examine the vibrations produced by a simple object : A cantilever beam • The vibrations depend on the impact position Spatialized haptic rendering
Simulation of vibrations (Euler-Bernouilli) • General solution : Euler Bernoulli model (EB) Spatialized haptic rendering
Simplified vibration patterns • Simplified patterns based on the physical behavior • Maybe easier perception ? • Simplified computation • Chosen model : exponentially damped sinusoid • Amplitude changes with impact position • Frequency changes with impact position • Both amplitude and frequency changes Spatialized haptic rendering
Simplified vibration patterns Am Fr AmFr (Consistent) AmCFr (Conflicting) Far impact Near impact
Evaluation • Objective: “Determine it is possible to perceive the impact position using vibration” • Population: 15 subjects • Apparatus : • Virtuose6D device • Sound blocking noise headphones Spatialized haptic rendering
Procedure • Task: Do two successive impacts. “Between these two impacts which one was the closest one from the hand ?” • 6 models • 2 realistic models (Euler-Bernoulli) (EB1, EB2) • 4 simplified models (Am, Fr, AmFr, AmCFr) • 4 impact positions • 8 random repetitions • Total of 576 trials (40 min) Spatialized haptic rendering
Results of quantitative evaluation • “How well was the subject able to determine the impact position by sensing the vibration ?” • Overall performance : • ANOVA Significant (p < 0.007) • Paired t-tests (p < 0.05) : • Am - • EB1 • EB2 • AmCFr • Fr - • EB1 • EB2 Spatialized haptic rendering Realistic Euler-Bernoulli Simplified
Results of qualitative evaluation • “How was the subjective feeling of realism ?” • Rate the impact realism : • Paired t-tests (p < 0.05): • Am • EB1 • EB2 • Fr • AmFr • EB2 • Fr • AmCFr Spatialized haptic rendering Realistic Euler-Bernoulli Simplified
Quantitative evaluation and inversion • Many participants inverted the interpretation of the vibrations Sensed Perceived Normal Inverted Spatialized haptic rendering
Discussion • Global weak inter – subject correlation : • Each subject seems to have his/her own interpretation (inversion or not) • Strong intra – subject consistency : • Subjects seem to be very consistent within his/her interpretation • Several strong inter – subject correlations between models • Several models interpreted the same way • Vibrations can be used to convey impact position information Spatialized haptic rendering
6DOF Spatialized haptic rendering • Generalize the previous result for 6DOF manipulation : • Virtual beam model Spatialized haptic rendering