Virtual Worlds: Input

1. Virtual Worlds: Input

2. Virtual Environments • Importance of immersion, presence, interaction, engagement, multisensory • Elements of the world: graphics, representation, visual and other senses, interface, navigation, manipulation, story • Interface: input, output, computer interface

3. Virtual Environments as a Medium Think of the following mediums and how they communicate: medium, kind of virtual world, final composition, how experienced Painting, dance, music, written word, play, movie, video game, virtual environments

4. Interaction in the Virtual World: Overview • User interface • Manipulation • Methods • Properties • Selection • Operations • Navigation • Wayfinding • Travel • Interaction and communication with others • Sharing • Collaboration

5. VR Worlds: Input Overview • User input: • 3D • Mouse, keyboard • Position tracking • Body tracking • Other physical input, including speech • World input

6. User Interface Metaphors • Familiar user interfaces: mobile phones, TV controls, remote controls, cars, video games, microwaves, doorways, cursor icons, desktop, touchscreens, notions of right, left and double clicking, mouseover, tweeting, texting • Do you use familiar ones in a VR world? Has limits and shortcuts

7. User Interface Interactions • Need to map user input to interactions • Users must already know how to use a tool or must be able to learn • Users bring experience and cultural knowledge • Users need familiarity but also want to extend what is normally possible in the real world (has limitations)

8. User Interface Interactions (con’t) • Viewpoint navigation (or travel) and object manipulation (and selection) are the main types of interaction • Differences between the two: • User’s conceptual model (what moves) • Extent of interaction space (navigation can be large, object manipulation often small) • Perceptual cues (navigation generally visual, object manipulation uses haptics)

9. 3D User Interfaces • Operate in 3D space • Natural: measure of how closely the actions in the virtual environment correspond to the actions in the real world • Examples: Wii Remote, Sony Move, Microsoft Kinect

10. Bowman, et all Questions about 3D User Interfaces and Naturalism • Are 3D inherently more natural than traditional? • Is naturalism the ultimate goal for designers? • Does naturalism lead to better performance, increased engagement, or better learning? • When naturalism isn’t possible should the designer go for traditional user interfaces or interfaces with some degree of naturalism?

11. Issues Involving Natural User Interfaces • For the tasks of navigation, selection and manipulation of object, how well do 3D natural user interfaces perform vs. magic techniques or some combination of them? • Issues involving turning, tracking, selecting, manipulating, steering • Natural interfaces sometimes out-perform and are sometimes inferior- depends on task and context; they need high level of accuracy and must be familiar

12. Definition of natural? • Lack of guidelines and standards for gestural interfaces • Problems raised by Norman and Nielsen of visibility, feedback, discoverability, consistency, scalability, reliability

13. Bowman et al table from Comm. of the ACM, 2012

14. Problems with Natural User Interfaces • How natural are they? • Who decides what the gestures or other actions will be? • Cultural differences

15. Input: Position Trackers • Position tracking – gives location and orientation of user and/or parts of user and feeds it to the computer - concerned with 6 degrees of freedom: x,y,z position and orientation, generally given by pitch, roll and yaw (angles with orthogonal axes) • General issues of accuracy (walking in a landscape vs surgery), latency (conducting an orchestra vs walking), interference from surrounding objects (noise, occlusion, monitors, metal), encumbrance such as cables (dancing, large-scale movement), cost

16. Position Trackers: Main Types • Optical • Electromagnetic • Mechanical • Tangible • Neural – brain • Others: 3D, Videometric, Ultrasonic, Inertial

17. Position Trackers: Optical • Visual information is used for tracking • Generally from video camera(s) • Generally need more than one unless don’t want 6 DOF; for example, the Kinect • Need good gesture recognition or visual recognition techniques (algorithms) • Sometimes use body sensors (motion capture) • Main disadvantage is line of sight issues

18. Electromagnetic Position Trackers • Very common • Need a transmitter (sends low-level magnetic fields from 3 orthogonal coils) and a receiver- can be in both configurations: transmitter on user or receiver on user • Strength of receiving signal varies with orientation and position • Some with cables and some wireless

19. Electromagnetic (con’t) • Advantages are that line of sight does not have to be clear • Disadvantages: can be interference from metal and monitors, must be within several feet of the transmitter • Eg. Ascension Flock of Birds, Polhemus

20. Position Trackers: Mechanical • Mechanical arms and booms that physically track movement • Eg. of BOOM by Fakespace • Advantages: accurate, no interference problems • Disadvantages: Can’t move very far

21. Position Trackers: Tangible • Uses hands in a more natural way: always been the case for tools from an axe to a toothbrush; relationship between design and interaction • Physical models (for architecture, urban planning) • Clay • Sand • Blocks • Interactive lighting devices • Use of effectors, LEDs, motors, sensors, react to heat • Boxes for manipulating music

22. Position Trackers: Neural • Field of brain-machine interface (BMI) • Attempts to read brain signals to direct the computer • Efforts to use EEGs, VEP (visually evoked potential), motor imagery • Neuroprosthetic devices, brain-implantable chips • Interfaces also benefit from research in neuroscience

23. Position Trackers: Others • Videometric: camera on person and tracks surroundings via landmarks such as infrared light sources • Ultrasonic: high-pitched sounds emitted at definite intervals – length of time – need multiple transmitters and receivers – subject to noise in the environment • Inertial: gyrocsopes, accelerometers, tracking device attached to user with cable- generally only orientation, so used with other devices- inexpensive, good accuracy • Neural: muscular (monitors electrical impulses in skin) – some experiments in reading brain wave

24. User Input: Body Tracking • Position of joints: Kinect • Head tracking- usually through orientation tracking- used to know what to project visually • Hand and fingers; position trackers, finger sensors (gloves), virtual scalpels • Torso and feet • Other body tracking such as heart rate, temperature • Indirect tracking using props and platforms

25. Body Input: Eye Tracking • Tracks movement of pupils – pretty accurate, requires that the head be still • eg. of Jeanne Stern’s project • Used for people with disabilities • Problem of “Midas Touch”

26. User Input: Other Physical Devices • Physical controls: buttons, switches, joysticks, mouse, steering wheels • Arduino devices • Props such as wands, 3D mouse, scalpel, drill, realistic devices for particular application (balls, bodies, spiders), pressure devices, mobile phones • Platforms: treadmills, locomotion, rings, kiosks, wheelchairs, cockpits, submarine control rooms, cars, workbenches, • CAVEs, DiVE system at Duke

27. Other Physical Input: Speech • Speech recognition: system will recognize commands, natural language interaction, Siri • Microphone • Problem of accents and training: speaker independence or dependence • How activated: button, command (talk), vision (need eye tracking) • Sometimes increases bandwidth by using separate processor

28. World Input to Virtual World • Persistent virtual worlds: exist over many experiences, multiple users, Web communication, databases • Importing data that changes (weather systems) • Importing real world data such as objects and obstacles, vision, knowledge, digital images, satellite info • Sometimes the info is brought in with transducers: translates data into electrical signals for the computer- devices such as microphones, weather stations, video, sensors, medical devices (for AR)

29. Sources Questioning Naturalism in 3D User Interfaces, Bowman, McMahan, and Ragan, Comm. Of the ACM, 2012 The Artificiality of Natural User Interfaces, Malizia and Bellucci, Comm of the ACM, 2012 Gestural Interfaces: A Step Backward In Usability, Norman and Nielsen, Interactions, 2010 Understanding Virtual Reality, Sherman & Craig, Morgan Kaufman, 2003 The Tangible User Interface and its Evolution, Ishi, Comm. Of the ACM, 2008 Neuroscience and the Future of Human-Computer Interface, Minnery & Fine, Interactions, Mar-Apr 2009 Building on Realism and Magic for Designing 3D Interaction Techniques by Kulik, IEEE Computer Graphics and Applications, Nov/Dec 2009

30. Sources – con’t Tangible Interaction=Form+Computing, Baskinger & Gross, ACM Interactions, Jan-Feb 2010 3D input devices, Frohlich et al, CG&A, Mar-Apr 2006 3D User Interfaces: New Perspectives and Directions, Bowman et al, Comp Graphics and Apps, Nov-Dec 2008 Usability of Multiple Degree-of-Freedom Input Devices and Virtual Reality: Displays for Interactive Visual Data Analysis, Moritz and Wischgoll, VRST (Virtual Reality Software and Technology) 2007