Interaction Techniques for Common Tasks in Immersive Virtual Environments

1. Interaction Techniques for Common Tasks in Immersive Virtual Environments Design, Evaluation, and Application Doug A. Bowman April 27, 1998

2. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Vision • Immersive VEs for productivity • Complex applications for real work • Example: immersive modeling and design

3. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Definitions • Interaction Technique (IT): Method used to complete a task via a human-computer interface (hardware & software) • Immersive VE: A real-time 3D synthetic environment that appears to surround the user in space • HMD with head tracking, CAVE • “Fishtank VR”, MUDs, Multimedia apps

4. Introduction Methodology Travel Selection/Manipulation Application Remaining Work A Brief History of VEs • 1968: Sutherland’s Ultimate Display • Hardware advances • displays – trackers – 3D graphics • input devices – haptics – 3D audio • Software advances • view culling – level of detail • VE toolkits – collision detection

5. Introduction Methodology Travel Selection/Manipulation Application Remaining Work VE Applications • In Use: • architectural walkthrough • phobia treatment • games (e.g. 1st person shooter) • Proposed: • information visualization and retrieval • modeling and design • constructivist education

6. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Interaction: the Distinguishing Factor • Current applications • may involve movement through VE • may involve shooting or pointing • Proposed applications • require 3D navigation and selection • require 6 DOF manipulation (object placement) • require large command spaces

7. Introduction Methodology Travel Selection/Manipulation Application Remaining Work How to improve VE Interaction • better design of techniques • systematic evaluation (formative and summative) • in the context of applications and requirements

8. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Universal Tasks: Travel • Viewpoint Motion Control: The user’s interactive control of the position and orientation of his viewpoint • Wayfinding: Cognitive process of determining a route, using landmarks, maps, etc. • Navigation: VMC + Wayfinding

9. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Universal Tasks: Selection & Manipulation • Selection: Specification of one or more objects from a set • as the object of a command • to begin manipulation • Manipulation: Specification of the position, orientation, and/or scale of an object

10. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Why not natural interaction? • Term “VR” implies replication of real world • Why not use well-developed human skills to accomplish tasks in VEs? • travel: walking or driving • selection & manipulation: grasp and place • These mappings are intuitive, but too limited

11. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Interaction Techniques and Input Devices • input devices are only the hardware component of an IT • input device does not determine IT • many ITs can be implemented with a single input device • we will not design or evaluate devices • we will design and evaluate ITs for common VE input devices

12. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Problem Statement: I will... • analyze universal tasks and create taxonomies of techniques • design new techniques based on these formal frameworks • design, implement, and conduct formal evaluations of IT performance • apply the results to a complex and useful VE application

13. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Design and Evaluation Methodology • Taxonomization and Categorization • Guided Design • Performance Measures • Range of Evaluation Methods • Testbed Evaluation

14. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Taxonomization and Categorization • Task analysis • Consider techniques for low-level subtasks • Promotes deeper understanding of task • Framework for design • Framework for evaluation Task Subtask Technique Component

15. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Guided Design • Design new techniques based on taxonomy, not simply intuition • Choose a component for each low-level subtask • Easy to see holes in design space 1 2 4 3 Task Subtask Technique Component

16. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Evaluation Methods • Range of performance metrics (quantitative and qualitative; productivity and user-centric) • Range of methods (user studies, usability evaluation, formal experiments) • Consideration of outside factors (characteristics of task, environment, user, system that might affect performance)

17. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Testbed Evaluation • testbed: representative set of tasks and environments • evaluate techniques for overall performance in a wide range of situations • vary technique components and outside factors • measure several performance variables • generalizable and replicable

18. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Summary of Methodology Initial Evaluation and Design Taxonomies Perf. Metrics Outside Factors environment density user’s reach task difficulty ... speed accuracy comfort ...

19. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Summary of Methodology Initial Evaluation and Design Taxonomies Perf. Metrics Outside Factors environment density user’s reach task difficulty ... speed accuracy comfort ... requirements Applications

20. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Summary of Methodology Initial Evaluation and Design Taxonomies Perf. Metrics Outside Factors environment density user’s reach task difficulty ... speed accuracy comfort ... TESTBED EVALUATION requirements Applications

21. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Summary of Methodology Initial Evaluation and Design Taxonomies Perf. Metrics Outside Factors environment density user’s reach task difficulty ... speed accuracy comfort ... TESTBED EVALUATION requirements Performance Measurements/ Models Applications choice of techniques

22. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Informal Evaluation • based on observations • default gaze-directed steering • lack of published work • based on our own applications • Conceptual Design Space • Virtual GIS

23. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Initial Taxonomy • Task: Move from the current location to the desired location gaze-directed pointing physical props Direction/Target Selection gesture slow in, slow out physical props Viewpoint Motion Control Velocity/Acceleration Selection start/stop buttons automatic start/stop constant movement Conditions of Input

24. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Performance Measures • Quantitative (e.g. speed, accuracy) • Qualitative (e.g. presence) • User-Centric (e.g. ease of use, comfort) Quality Factors -speed -accuracy -cognitive load -presence -spatial awareness - ... IT Apps

25. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Simple Experiments(Bowman, Koller, and Hodges, VRAIS ‘97) • Absolute Motion • no difference between gaze and pointing • Relative Motion • pointing superior to gaze • Spatial Awareness • teleportation causes disorientation • any continuous motion does not

26. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Expanded Framework • Absolute vs. Relative Motion • same techniques, different results • highlights need to consider outside factors • Consider task, user, system, and environment characteristics Performance absolute relative Task gaze-directed steering pointing

27. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Complex Experiment • Does travel IT affect cognitive load? • Task: gather as much info as possible • Variables: • IT: gaze, pointing, torso • Environment: 1-, 2-, or 3-dimensional • System: collision detection

28. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Guided Design • Taxonomy: “tour” technique • environmental target selection • gesture-based velocity selection • explicit or automatic stop inputs • Intuition: travel based on manipulation • cross-task technique • still fits in taxonomy

29. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Final Framework and Testbed • Rework taxonomy to be more general • task analysis: 2 basic position-setting methods are specifying destination, specifying trajectory • distinction allows better fitting of techniques • VMC Testbed • still in design stage • based on evaluation framework

30. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Initial Taxonomy • Based on metaphor, not task • Arm-extension metaphor • touch and place object with virtual hand • hand may extend beyond normal range • Ray-casting metaphor • point at object to select • manipulate by attaching to virtual light ray

31. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Informal Evaluation(Bowman and Hodges, I3DG ‘97) • Studied six techniques (4 AE, 2 RC) • Simple user study (comments, observations) • Eleven subjects used techniques to place furniture in a room • Results • AE excels at manip., RC better at selection • selection & manipulation should be separated

32. Introduction Methodology Travel Selection/Manipulation Application Remaining Work HOMER Techniques • Hand-Centered Object Manipulation Extending Ray-Casting • Hybrid technique • Select: ray-casting • Manipulate: v. hand Time

33. Introduction Methodology Travel Selection/Manipulation Application Remaining Work HOMER Techniques • Hand-Centered Object Manipulation Extending Ray-Casting • Hybrid technique • Select: ray-casting • Manipulate: v. hand Time

34. Introduction Methodology Travel Selection/Manipulation Application Remaining Work HOMER Techniques • Hand-Centered Object Manipulation Extending Ray-Casting • Hybrid technique • Select: ray-casting • Manipulate: v. hand Time

35. Introduction Methodology Travel Selection/Manipulation Application Remaining Work HOMER Techniques Hand-Centered Object Manipulation Extending Ray-Casting • Hybrid technique • Select: ray-casting • Manipulate: v. hand Time

36. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Formal Taxonomy touch occlude Indication of Object Selection button gesture Indication to Select hand moves to object user scales to touch object Attachment Manipulation 1-to-1 hand motion mapping Positioning match tracker orientation indirect control Orientation button gesture Indication to Release Release Final Object Position/Orientation remain in place use physics model

37. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Evaluation Framework • Performance measures similar to travel • Important outside factors: • task characteristics: DOFs to manipulate • user characteristics: reach, spatial ability • system characteristics: constraints used

38. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Guided Design • testbed implemented to allow arbitrary combinations of technique components • 4608 possible combinations - reduced to 667 via dependencies and constraints • Taxonomy: gaze-based HOMER with separate positioning and orientation • Intuition: manipulation based on travel (cross-task technique)

39. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Selection/Manipulation Testbed • Tasks that test all important aspects of a select/manip. IT • Selection variables: distance, size, density • Manip. variables: distance, accuracy, DOFs required

40. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Application Case Study: Immersive Design • Verify evaluation results in a complex VE application • Design system involves all universal tasks • Choose ITs based on testbed results and specified application requirements

41. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Interaction Requirements • Travel • exploration and goal-based movements • spatial awareness, info gathering, ease of use • Selection • accuracy at a distance, speed, comfort • Manipulation • expressibility, accuracy, ease of use

42. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Three Levels of Interaction Design • Naive design • taken from CDS application (in D. Bertol, Designing Digital Space) • gaze-directed steering, ray-casting • “Intuitive” design iteration • current implementation • pen & tablet, pointing, Go-Go technique • Final, systematic design

43. Introduction Methodology Travel Selection/Manipulation Application Remaining Work Remaining Work • complete design and evaluation framework for travel • design, implement, and run travel testbed • complete and run selection/manipulation testbed • modify application interaction design and verify with a usability study

44. Contributions • formal understanding of tasks/techniques • testbeds for future evaluations • performance results and models • new interaction techniques • useful and usable immersive design application