Interacting With Dynamic Real Objects in a Virtual Environment

1. Interacting With Dynamic Real Objects in a Virtual Environment Benjamin Lok February 14th, 2003

2. Why we need dynamic real objects in VEs How we get dynamic real objects in VEs What good are dynamic real objects? Applying the system to a driving real world problem Outline • Motivation • Incorporation of Dynamic Real Objects • Managing Collisions Between Virtual and Dynamic Real Objects • User Study • NASA Case Study • Conclusion

3. Assembly Verification • Given a model, we would like to explore: • Can it be readily assembled? • Can repairers service it? • Example: • Changing an oil filter • Attaching a cable to a payload

4. Current ImmersiveVE Approaches • Most objects are purely virtual • User • Tools • Parts • Most virtual objects are not registered with a corresponding real object. • System has limited shape and motion information of real objects.

5. Ideally • Would like: • Accurate virtual representations, or avatars, of real objects • Virtual objects responding to real objects • Haptic feedback • Correct affordances • Constrained motion • Example: Unscrewing a virtual oil filter from a car engine model

6. Dynamic Real Objects • Tracking and modeling dynamic objects would: • Improve interactivity • Enable visually faithful virtual representations • Dynamic objects can: • Change shape • Change appearance

7. Thesis Statement Naturally interacting with real objects in immersive virtual environments improves task performance and presence in spatial cognitive manual tasks.

8. Previous Work: Incorporating Real Objects into VEs • Non-Real Time • Virtualized Reality (Kanade, et al.) • Real Time • Image Based Visual Hulls [Matusik00, 01] • 3D Tele-Immersion [Daniilidis00] • Augment specific objects for interaction • Doll’s head [Hinkley94] • Plate [Hoffman98] • How important is to get real objects into a virtual environment?

9. Previous Work: Avatars • Self - Avatars in VEs • What makes avatars believable? [Thalmann98] • What avatars components are necessary? [Slater93, 94, Garau01] • VEs currently have: • Choices from a library • Generic avatars • No avatars • Generic avatars > no avatars [Slater93] • Are visually faithful avatars better than generic avatars?

10. Visual Incorporation of Dynamic Real Objects in a VE

11. Motivation • Handle dynamic objects (generate a virtual representation) • Interactive rates • Bypass an explicit 3D modeling stage • Inputs: outside-looking-in camera images • Generate an approximation of the real objects (visual hull)

12. … … Reconstruction Algorithm 1. Start with live camera images 2. Image Subtraction 3. Use images to calculate volume intersection 4. Composite with the VE

13. Visual Hull Computation • Visual hull - tightest volume given a set of object silhouettes • Intersection of the projection of object pixels

14. Visual Hull Computation Visual hull - tightest volume given a set of object silhouettes Intersection of the projection of object pixels

15. Volume Querying • A point inside the visual hull projects onto an object pixel from each camera

16. Implementation • 1 HMD-mounted and 3 wall-mounted cameras • SGI Reality Monster – handles up to 7 video feeds • Computation • Image subtraction is the most work • ~16000 triangles/sec, 1.2 gigapixels • 15-18 fps • Estimated error: 1 cm • Performance will increase as graphics hardware continues to improve

17. Results

18. Managing Collisions Between Virtual and Dynamic Real Objects

19. Approach • We want virtual objects respond to real object avatars • This requires detecting when real and virtual objects intersect • If intersections exist, determine plausible responses

20. Assumptions • Only virtual objects can move or deform at collision. • Both real and virtual objects are assumed stationary at collision. • We catch collisions soon after a virtual object enters the visual hull, and not as it exits the other side.

21. Detecting Collisions

22. Resolving Collisions Approach • Estimate point of deepest virtual object penetration 2. Define plausible recovery vector 3. Estimate point of collision on visual hull

23. Resolving Collisions Approach • Estimate point of deepest virtual object penetration 2. Define plausible recovery vector 3. Estimate point of collision on visual hull

24. Resolving Collisions Approach • Estimate point of deepest virtual object penetration 2. Define plausible recovery vector 3. Estimate point of collision on visual hull

25. Resolving Collisions Approach • Estimate point of deepest virtual object penetration 2. Define plausible recovery vector 3. Estimate point of collision on visual hull

26. Resolving Collisions Approach • Estimate point of deepest virtual object penetration 2. Define plausible recovery vector 3. Estimate point of collision on visual hull

27. Results

28. Results

29. Collision Detection / Response Performance • Volume-query about 5000 triangles per second • Error of collision points is ~0.75 cm. • Depends on average size of virtual object triangles • Tradeoff between accuracy and time • Plenty of room for optimizations

30. Spatial Cognitive Task Study

31. Study Motivation • Effects of • Interacting with real objects • Visual fidelity of self-avatars • On • Task Performance • Presence • For spatial cognitive manual tasks

32. Spatial Cognitive Manual Tasks • Spatial Ability • Visualizing a manipulation in 3-space • Cognition • Psychological processes involved in the acquisition, organization, and use of knowledge

33. Hypotheses • Task Performance: Participants will complete a spatial cognitive manual task faster when manipulating real objects, as opposed to virtual objects only. • Sense of Presence: Participants will report a higher sense of presence when their self-avatars are visually faithful, as opposed to generic.

34. Task • Manipulated identical painted blocks to match target patterns • Each block had six distinct patterns. • Target patterns: • 2x2 blocks (small) • 3x3 blocks (large)

35. Measures • Task performance • Time to complete the patterns correctly • Sense of presence • (After experience) Steed-Usoh-Slater Sense of Presence Questionnaire (SUS) • Other factors • (Before experience) spatial ability • (Before and after experience) simulator sickness

36. Conditions Purely Virtual • All participants did the task in a real space environment. • Each participant did the task in one of three VEs. Real Space Hybrid Vis. Faithful Hybrid

37. Conditions Sense of presence Task performance

38. Real Space Environment • Task was conducted within a draped enclosure • Participant watched monitor while performing task • RSE performance was a baseline to compare against VE performance

39. Purely Virtual Environment • Participant manipulated virtual objects • Participant was presented with a generic avatar

40. Hybrid Environment • Participant manipulated real objects • Participant was presented with a generic avatar

41. Visually-Faithful Hybrid Env. • Participant manipulated real objects • Participant was presented with a visually faithful avatar

42. Task Performance Results

43. Task Performance Results * - significant at the =0.05 level ** - =0.01 level *** - =0.001 level

44. Sense of Presence Results

45. Sense of Presence Results

46. Debriefing Responses • They felt almost completely immersed while performing the task. • They felt the virtual objects in the virtual room (such as the painting, plant, and lamp) improved their sense of presence, even though they had no direct interaction with these objects. • They felt that seeing an avatar added to their sense of presence. • PVE and HE participants commented on the fidelity of motion, whereas VFHE participants commented on the fidelity of appearance. • VFHE and HE participants felt tactile feedback of working with real objects improved their sense of presence. • VFHE participants reported getting used to manipulating and interacting in the VE significantly faster than PVE participants.

47. Study Conclusions • Interacting with real objects provided a quite substantial performance improvement over interacting with virtual objects for cognitive manual tasks • Debriefing quotes show that the visually faithful avatar was preferred, though reported sense of presence was not significantly different. • Kinematic fidelity of the avatar is more important than visual fidelity for sense of presence. Handling real objects makes task performance and interaction in the VE more like the actual task.

48. Case Study: NASA Langley Research Center (LaRC)Payload Assembly Task

49. NASA Driving Problems • Given payload models, designers and engineers want to evaluate: • Assembly feasibility • Assembly training • Repairability • Current Approaches • Measurements • Design drawings • Step-by-step assembly instruction list • Low fidelity mock-ups

50. Task • Wanted a plausible task given common assembly jobs. • Abstracted a payload layout task • Screw in tube • Attach power cable