3D Interaction Techniques for Virtual Environments

1. 3D Interaction Techniques for Virtual Environments Doug A. Bowman Edited by C. Song

2. Technique Classificationby metaphor

3. Manipulation metaphors I • Ray casting • little effort required • Exact positioning and orienting very difficult (lever arm effect)

4. 5.4.2 Interacting by Pointing • Selection process • When the vector defined by the direction of pointing intersects a virtual object, the user can select it by triggering event that confirms the selection. • Examples of triggers are buttons and voice command • A number of pointing techniques have been reported • How the pointing direction is defined • By the shape of the selection volume • Pointing is a powerful selection technique • Pointing, however, is generally a very poor positioning (Manipulation) technique • Rotation can be effectively accomplished only about one axis. • Expressive 6-DOF manipulation is therefore impossible

5. 5.4.2 Interacting by Pointing • Simple ray-casting technique • Two-handed pointing • Flashlight / Aperture techniques • Image-plane techniques • Fishing-reel techniques

6. Ray Casting (I) • The user points at objects with a virtual ray that defines the direction of pointing. • Attached to the virtual hand controlled by a 6-DOF sensor in immersive environment. • Attached to a 3D widget controlled by a mouse in a desktop 3D environment. • Pointing direction • p() = h +  P, where P = the direction of user’s virtual hand, h= 3D position of the virtual hand • More than one object can be intersected by the line p(), and only one closest to the user should be selected.

7. Ray Casting (II) • The shape of the ray • Short line segment; Figure 5.4 • Infinitely long virtual ray; better visual feedback • Virtual ray casting is a very powerful selection technique • Except when a very high precision of selection is required; selecting small or faraway objects • At close range, ray-casting is perhaps the most simple and efficient selection technique.

8. Two-Handed Pointing • Two handed technique: one hand specifies the origin of the virtual ray, while the other hand specifies where the ray is pointing to • p() = hl +  ( hr – hl ) , where hr and hl = 3D position of the right and left virtual hands respectively. • Disadvantage • Both hands must be tracked. • Advantage • Allow for richer and more effective pointing interaction • Curve the virtual pointer by twisting the hand slightly. • Fig. 5.5 참조.

9. Flashlight and Aperture Techniques (I) • Spotlight or flashlight technique • It replaces the virtual ray with a conic selection volume, with the apex of the cone at the input device; Fig. 5.6 • Objects that fall within this selection cone can be selected. • Easy selection of small objects even when they are located far from the user • Disambiguation • When more than one object falls into the spot light. • Disadvantages: • When selection of small objects or tightly grouped objects

10. Flashlight and Aperture Techniques (II) • Aperture Technique • Modification of the flashlight technique and improve it. • Selection line P() p() = e +  (h – e ) , e = virtual viewpoint h = hand • The user can interactively control the spread angle of the selection volume simply by bringing the hand sensor closer or moving it father away; • Figure 5.6 • Simplify selection of virtual objects by using the orientation of the pointer around a central avis as an additional disambiguation metric. • Figure 5.7b • Used easily in both desktop and immersive VEs.

11. Image-Plane Techniques • The user selects and manipulates 3D objects by touching and manipulating their 2D projections on a virtual image plane located in front of the user. • Figure 5.8 • Sticky-finger technique • Figure 5.8 • Head-crusher technique • Another Image-plane tech. With a data glove devices • With two fingers, his thumb and index fingers • Allows the user to modify the orientation of 3D objects, but their distance from the user can not be directly controlled. • Mine’s scaled-world grab, Pierce’s Voodoo Doll technique

12. Fishing-Reel Technique • The ray direction is controlled by the spatial movements of user’s hand, • While distance is controlled by other means. • Controlling the length of the virtual ray. • A simple mechanical slider or a pair of buttons added to the tracking device

13. Manipulation metaphors II • Simple virtual hand • Natural but limited

14. 5.4.3 Direct Manipulation: Virtual Hand Techniques • 3D cursor • 3D model of human hand • Semitransparent volumetric cursor • Selection • The user intersects 3D cursor with the target of selection • Trigger technique : to pick it up • Button press, voice command, hand gesture • The object is attached to the virtual hand and can be easily translated and rotated within V.E • Release • The user release it with another trigger.

15. Virtual Hand Interaction Techniques • Simple (“Classical” ) virtual hand technique • Go-Go technique • Indirect Go-Go

16. Simple Virtual Hand (I) • Direct mapping : • transfer functions or control-display gain function • pv = αpr , Rv = Rr (Eq. 5.4) • pr Rr are the position and orientation(3*3 matrix) of the user’s real hand. • pv , Rv are the corresponding position and orientation of the virtual hand • α is a scaling ratio to match the scales of the real and virtual coordinate systems.

17. Simple Virtual Hand (II) • Scaling rotation • It is useful to “scale” 3D device rotations similar to the way we scale translations • Fundamental problem • In order to select objects located further away, the user must employ a travel technique. Inconvenient and increase the complexity of the 3D UI.

18. Go-Go Interaction Technique While the user’s real hand is close to the user (the distance to the hand is smaller than threshold D), the mapping is one to one, the movement of the virtual hand correspond to the real hand movements As the user extends her hand beyond D, the mapping becomes nonlinear and the virtual arm “grows”, thus permitting the user to access and manipulate remote objects. C1 continuity is ensured.

19. Go-Go IT • Different mapping function can be used to achieve a different control-display gain between the real and virtual hands. • Advantages: • Provide direct, seamless, 6-DOF object manipulation both close to the user and at a distance. • Disadvantages: • As the distance increases, the technique maps small movements of the user’s hand into large movements of the virtual hand, which complicates precise positioning at a distance.

20. 5.4.5 Combining Techniques • Aggregation of techniques • Mechanism for choosing the desired manipulation technique from a limited set of possible options • Technique integration • Combining techniques in which the interface switches transparently between interaction techniques depending on the current task context. • Selection과 manipulation 이 반복 적용되므로, 각 mode에서 최선의 방법들을 선택하여 적용한다.

21. HOMER technique Hand-Centered Object Manipulation Extending Ray-Casting • Select: ray-casting • Manipulate: hand Time

22. Manipulation metaphors III • HOMER (ray-casting + arm-extension) • Easy selection & manipulation • Expressive over range of distances • Hard to move objects away from you

23. HOMER implementation • Requires torso position t • Upon selection, detach virtual hand from tracker, move v. hand to object position in world CS, and attach object to v. hand (w/out moving object) • Get physical hand position h and distance dh = dist(h, t) • Get object position o and distance do = dist(o, t)

24. HOMER implementation (cont.) • Each frame: • Copy hand tracker matrix to v. hand matrix (to set orientation) • Get physical hand position hcurrand distance: dh-curr = dist(hcurr, t) • V. hand distance • Normalize torso-hand vector • V. hand position vh = t + dvh*(thcurr)

25. Manipulation metaphors IV • World-in-miniature (WIM) • All manipulation in reach • Doesn’t scale well, indirect • Scaled-world grab • Easy, natural manipulation • User discomfort with use

26. 5.4.4 World-in-miniature (WIM) • “Dollhouse” world held in user’s hand • Miniature objects can be manipulated directly • Moving miniature objects affects full-scale objects

27. World-in-Miniature (WIM) • Provides the user with miniature handheld model of VE, which is an exact copy of the VE at a small scale. • Careful use of back face culling techniques. Only the “inside” of the walls of the room model should be rendered. • WIM combine navigation with manipulation. • Although WIM works well for small and medium-sized environments, using WIM in a very large environment would require an extreme scale factor, resulting in very small object copies in the WIM. This would make accurate selection and manipulation extremely difficult. • It has been successfully used in 3D interfaces for Augmented reality, in desktop 3D Uis. In fact, it can be considered a 3D generalization of the traditional overview maps that are often used in 3D games.

28. Root head hand room Root table head hand room WIM room (scaled) table table copy WIM implementation

29. Scale-world Grab • Based on HOMER • Selection : Image-plane selection tech. is used. • Manipulation : • Scale down the entire VE around the user’s virtual viewpoint switching into manipulation mode • In HOMER, scaling the user’s hand motion • The scaling coefficient s = Dv / Do where Dv is the distance from the virtual viewpoint to the virtual hand, and Do is the distance from the virtual viewpoint to the selected object. • Well for operations at a distance but not effective when the user wants to pick up an object located within arm’s reach and move it farther away.

30. Voodoo Dolls • To overcome the scaling approach such as HOMER, scaled-World grab when the user needs to move local objects farther away. • Image-plane and WIM with a pair of pinch gloves • Manipulate virtual objects indirectly using miniature handheld copies of objects called “dolls”; Figure 5.13 • Selecting the target object with an image-plane technique, which creates the dolls representing the target objects and places them in the user’s hand. • The doll in her non-dominant hand : the corresponding virtual object does not move when the user moves this doll. • To start manipulation, the user simply passes the doll into dominant hand.

31. Voodoo Dolls • Advantages: • Powerful IT allowing to perform some sophisticated tasks, such as the manipulation of moving, animated objects • Key idea is very universal and important insight • Separating functionality depending on the dominant and non-dominant hands • Disadvantages • Increases H/W demand • Direct application to a desktop might be difficult.

32. Technique Classification by metaphor

33. Technique Classificationby components

34. Evaluation: positioning task • Ray casting effective if the object is repositioned at constant distance • Scaling techniques (HOMER, scaled world grab) difficult in outward positioning of objects: e.g. pick an object located within reach and move it far away • If outward positioning is not needed then scaling techniques might be effective

35. Evaluation: orientation task • Setting precise orientation can be very difficult • Shape of objects is important • Orienting at-a-distance harder than positioning at-a-distance • Techniques should be hand-centered

36. Manipulation notes • No universally best technique • Constraints and reduced DOFs • Naturalism not always desirable • If VE is not based in the real, design it so manipulation is optimized

37. Manipulation enhancements • Constraints • 2-handed manipulation • Haptic feedback • Multi-modal manipulation

38. Implementation issues for manipulation techniques • Integration with selection technique • Disable selection and selection feedback while manipulating • What happens upon release?

39. SIGGRAPH 2001 for Manipulation • 참고할 것.

40. Common manipulation techniques • Simple virtual hand • HOMER • Scaled-world grab • World-in-miniature

41. Simple virtual hand technique Root • Attach object to virtual hand, by making object a child of the hand (w/out moving object) • On release, reattach object to world (w/out moving object) • Also applies to Go-Go (and other arm-extension techniques) and ray-casting head hand building Root head hand building

42. 2.0 m 1.0 m 0.6 m 0.3 m torso torso physical hand physical hand HOMER technique Time Hand-Centered Object Manipulation Extending Ray-Casting • Select: ray-casting • Manipulate: hand

43. HOMER implementation • Requires torso position t • Upon selection, detach virtual hand from tracker, move v. hand to object position in world CS, and attach object to v. hand (w/out moving object) • Get physical hand position h and distance dh = dist(h, t) • Get object position o and distance do = dist(o, t)

44. HOMER implementation (cont.) • Each frame: • Copy hand tracker matrix to v. hand matrix (to set orientation) • Get physical hand position hcurrand distance: dh-curr = dist(hcurr, t) • V. hand distance • Normalize torso-hand vector • V. hand position vh = t + dvh*(thcurr)

45. Scaled-world grab technique • Often used w/ occlusion • At selection, scale user up (or world down) so that v. hand is actually touching selected object • User doesn’t notice a change in the image until he moves

46. Scaled-world grab implementation • At selection: • Get world CS distance from eye to hand deh • Get world CS distance from eye to object deo • Scale user (entire user subtree) uniformly by deo / deh • Ensure that eye remains in same position • Attach selected object to v. hand (w/out moving object) • At release: • Re-attach object to world (w/out moving object) • Scale user uniformly by deh / deo • Ensure that eye remains in same position

47. World-in-miniature (WIM) technique • “Dollhouse” world held in user’s hand • Miniature objects can be manipulated directly • Moving miniature objects affects full-scale objects • Can also be used for navigation

48. Root head hand room Root table head hand room WIM room (scaled) table table copy WIM implementation

49. WIM implementation (cont.) • On selection: • Determine which full-scale object corresponds to the selected miniature object • Attach miniature object to v. hand (w/out moving object) • Each frame: • Copy local position matrix of miniature object to corresponding full-scale object

50. Implementation issues for manipulation techniques • Integration with selection technique • Disable selection and selection feedback while manipulating • What happens upon release?