Direct Manipulation Interfaces

1. Direct Manipulation Interfaces • Positive acceptance of an application • Mastery of the interface • Competence in performing tasks • Ease in learning originally and in assimilating advanced features • Confidence in the capacity to retain mastery over time • Enjoyment in using the interface • Eagerness to show off interfaces to novices • Desire to explore more powerful aspects

2. Direct Manipulation Interfaces • Attributes of direct-manipulation interfaces • Visibility of the objects and actions of interest • Example: Driving an automobile • Rapid, reversible, incremental actions • Replacement of types commands by a pointing action on the object of interest • Example: Dragging a file to a trash can versus “rm file.doc” • What about: “rm file*.doc”? • What about “rm *.*” with no undo?

3. Direct Manipulation Interfaces • Extensions of direct manipulation • Virtual reality – users are in an immersed environment • Reality is blocked out via a head-mounted display • Hand gestures (via a data glove) allow users to point, select, grasp and navigate • Augmented reality – user remains in normal surroundings, but adds a transparent overlay. • Examples – labeled buildings, hidden plumbing • Google: Project Glass • http://www.youtube.com/watch?v=W2E2zcFt9Xo • http://www.youtube.com/watch?v=tnRJaHZH9lo&feature=related • Tangible user interfaces – users manipulate physical objects • Example – putting several plastic blocks near each other to create an office floor plan

4. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • The Tubeless Interface

5. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • Single-line and Multi-line Editors (e.g., IBM MVS, VM, TSO, JCL)

6. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • Single-line and Multi-line Editors versus WYSIWYG (what you see is what you get) editors (e.g., Microsoft Word – early 1990s) • Example: Three basic modes of vi • Command mode (Telling the computer what to do: Low level commands, e.g., move the cursor to the right one character) • Default when you enter vi. • Most letters, or short sequences of letters, that you type will be interpreted as commands • Pressing Esc when you're in command mode, your terminal will beep at you. This is a very good way to tell when you're in command mode • Insert mode (Entering the content) • Whatever you type is inserted in the file at the cursor position • Press Esc to end insert mode, and return to command mode. • Line mode (Telling the computer what to: High level commands, e.g. Save) • To enter line mode from command mode, type a colon ( : ) • Your cursor moves to the bottom of the screen, by a colon prompt. • Type a line mode command, then press Enter.

7. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • Single-line and Multi-line Editors (e.g., vi) Starting vi and Saving Files Starting vi: vi filename (start editing filename, create it if necessary) Saving the file you're working on and/or leaving vi: :wq (write the file to disk and quit) Quit without saving any changes: :q! :w! newfile (write all lines from the entire current file into the file 'newfile', overwriting any existing newfile) :n,m w! newfile (write the lines from n to m, inclusive, into the file newfile, overwriting any existing newfile)

8. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • Single-line and Multi-line Editors

9. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • Single-line and Multi-line Editors • Searching for text • Inserting text

10. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • Single-line and Multi-line Editors • Deleting text

11. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • Single-line and Multi-line Editors • Cutting and Pasting • Miscellaneous Commands

12. Direct Manipulation Interfaces • Command-line versus display editors versus word processors • Early 1980s – Text editing was done with line-oriented command languages • Nroff/troff: Unix based word processor .nf = no-fill, you use for graphs or text that you don't want spaces to be ignored .ce # = centers by the # of sentences you enter .ti # = .ti indents but only for one line, so if you have one sentence that needs to be indented 8 spaces but the rest of the page is indented 3 you can use .ti 8 for that single sentence .fi = fill-in, extra space will be ignored and text that is entered like this, the fill-in command will continue until you enter a .nf command and vice-versa .in # = # is the number of spaces you want the text indented, this command will be in place until you put in a new .in command...if you want something to not be indented you put in .in 0

13. Direct Manipulation Interfaces • Command-line editors • Who would use these interfaces? • Why?

14. Direct Manipulation Interfaces • Command-line editors • Who would use these interfaces? • Why? • Mastery of the interface • Competence in performing tasks • Ease in learning originally and in assimilating advanced features • Confidence in the capacity to retain mastery over time • Enjoyment in using the interface • Eagerness to show off interfaces to novices • Desire to explore more powerful aspects

15. Direct Manipulation Interfaces • Command-line editors • Who would use these interfaces? • Why? • BETTER THAN A TYPEWRITER • BETTER THAN A SLIDE RULE • BETTER THAN A CALCULATOR

16. Direct Manipulation Interfaces • Advantages of WYSIWYG Editors • Users see a full page of text • 20 to 60 lines provides a context for each sentence • The document is seen as it will appear when printed • Eliminating the clutter of formatting commands • Cursor action is visible • Indicates where to focus attention and apply action • Cursor motion is natural • Arrow keys or mouse provide natural physical mechanisms for moving the cursor • Labeled icons make frequent actions rapid • Toolbar for frequent actions • Immediate display of the results of an action • Example: Clicking a button to center text provides immediate result • Rapid response and display • Full page of text in a fraction of a second • Easily reversible actions • Example: Undo, backspace

17. Direct Manipulation Interfaces • Technology advancements evolving from word processing • Integration of graphics, spreadsheets, animations, photographs, etc. • Desktop publishing software • Presentation software • Hypermedia environments and the World Wide Web (hyperlinks to documents) • Improved macro facilities (e.g., construct, save and edit sequences of frequently used actions) • Spell checkers and thesauri • Grammar checkers • Use of passive voice • Excessive use of certain words • Lack of parallel construction • Document assemblers • Contracts • Wills

18. Direct Manipulation Interfaces • Spreadsheets • 1979 – VisiCalc from a Harvard Business School student • 254 rows and 63 columns • Functions within a cell as it relates to other cells • Simulation of an accountants spreadsheet • Lotus 1-2-3 dominated the market in the 1980s • Today Excel dominates • Graphics displays • Multiple windows • Statistical routines • Database access (e.g., Price List to Service Catalog)

19. Direct Manipulation Interfaces • Spatial Data Management • Geographic applications – Nicholas Negroponte at MIT • Display of the world and able to zoom in on the Pacific Ocean to see markers for convoys of military ship

20. Direct Manipulation Interfaces • Spatial Data Management • ArcView – ESRI • Global Information Systems (Demo) • Select type of information to display (roads, population, rainfall, topography, political boundaries

21. Direct Manipulation Interfaces • Video Games • The most exciting, well-engineered, commercially successful application of direct-manipulation concepts? • Pong • Pacman (http://www.activitypad.com/online-games/pacman/) • Field of action is visual and compelling • Button presses, joystick motions and knob rotations produce rapid response on the screen • No syntax to remember • Error messages are rare – the results of the action are obvious and easily reversed • Often there is continuous display of the score (competition between others and the player themselves) • Positive reinforcement that encourages mastery

22. Direct Manipulation Interfaces • Video Games • Educational Video Games – direct manipulation • SimCity – education on urban planning • The Sims – stronger attraction to women then men

23. Direct Manipulation Interfaces • Video Games • Computer Role Playing Games (CRPGs) • Players assume the role of a fictional character • Activity takes place in a fictional world • User’s control many of their character's actions • Myst • Massively Multiplayer On-Line Role Playing Games (MMORPGs) • A large number of players interact with one another in a virtual world • Interaction is in a persistent world • hosted by the game's publisher • continues to evolve while the player is away from the game • Worldwide MMORPGs revenues exceeded half a billion dollars in 2005 • World of Warcraft

24. Direct Manipulation Interfaces • Video Games versus Business Applications • Game players • Engaged in competition with a system or other players • Seek entertainment and focus on challenge • May prefer random events • Application users • Prefer a strong internal locus of control • Focus on their tasks and may resent too many playful distractions • Do not prefer random events

25. Direct Manipulation Interfaces • Computer-aided Design • Automobiles, electronic circuitry, aircraft, mechanical engineering • Structural engineering, floorplans, interiors, landscaping, plumbing, electrical installation, etc. • When the design is complete, the program can provide information regarding: • Current • Voltage drops • Fabrication costs • Manufacturing problems

26. Direct Manipulation Interfaces • Computer – aided Manufacturing and Process Control • Honeywell’s Experion Process Knowledge System • Provides the manager of a oil refinery or power utility plant with a colorized schematic of the plant • Can indicate with a red line a sensor value that is out of range • With a single click the operator can get a more detailed view of the troubling component • A second click can provide more detailed information the sensor, or reset a value or circuit • Basic strategy: eliminate the need for complex commands that the operator might only need to recall during a once-a-year emergency

27. Direct Manipulation Interfaces • Direct Manipulation in Office Automation • Xerox Star • Sophisticated Text Formatting • Graphics • Multiple Fonts • High Resolution • Cursor Based Interface • Apple Lisa • Precursor to the Macintosh • Hardware and software designs supported • Pull-down menus • Multi-window manipulation • Editing of graphics and text • Dragging of icons

28. Direct Manipulation Interfaces • Direct Manipulation in Office Automation • MS-DOS Commands vs. Macintosh Direct Manipulation • Tasks: Creating, copying, renaming, erasing files • After training and practice, average task times: • MS-DOS is 5.8 minutes • Macintosh is 4.8 minutes • After training and practice, average errors: • MS-DOS is 2.0 • Macintosh is 0.8

29. Direct Manipulation Interfaces • Continuing evolution of direct manipulation • Quicken • Home Automation • Direct manipulation on a floor plan of: • Burglar alarms • Heat sensors • Smoke detectors • Opening/closing curtains or screens • Air conditioning and heating • Audio/video speakers or screens • E.g., users can route sound from a MP3 player located in the living room to the kitchen by dragging the MP3 icon into the kitchen • Virtual Worlds • Travel through the human body • Ride an electron cloud as it spins around a nucleus

30. Direct Manipulation Interfaces • Continuing evolution of direct manipulation • Problems with direct manipulation • Spatial or visual representation are not necessarily an improvement over text • Especially for blind or visually impaired users • Direct manipulation designs may consume considerable screen space • May result in scrolling or multiple actions • Users must learn the meanings of visual representations • Icon interpretation • For experienced typists, taking a hand off the keyboard to move a mouse may take more time then typing the relevant command • Users may not share the same understanding of the metaphor, analogy, or conceptual model with the designer (testing is required) • Browser based applications limit direct manipulation (e.g., drag & drop) • Require Dynamic HTML, Java or Flash

31. Direct Manipulation Interfaces • Advantages of direct manipulation • Continuous representation of the objects and actions of interest with meaningful visual metaphors • Physical actions or presses of labeled buttons, instead of complex syntax • Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately

32. Direct Manipulation Interfaces • Advantages of direct manipulation • Design systems with the following benefits • Novices can learn basic functionality quickly • Experts can work rapidly to carry out a wide range of tasks, even defining new functions and features • Knowledgeable intermittent users can retain operational concepts • Error messages are rarely needed • Users can immediately see whether their actions are furthering their goals, and if the actions are counterproductive, they can change the direction of their activity • Users experience less anxiety because the interface is comprehensible and because actions are easily reversed • Users gain confidence and mastery because they are the initiators of action, they feel in control, and they predict the interface’s responses

33. Direct Manipulation Interfaces • The OAI Model and Direct Manipulation • The object of interest is displayed so that interface actions are close to the high-level task domain • Little need for mental decomposition of tasks into multiple interface commands with complex syntactic forms (e.g., vi: go to line, go to word, go to character) • Each action produces a comprehensible result in the task domain that is visible in the interface immediately • The closeness of the task domain to the interface domain reduces operator problem-solving load and stress • Compared to textual descriptors, visual representations of objects may be more natural and closer to human innate capabilities • Action and visual skills emerged well before language in human evolution

34. Direct Manipulation Interfaces • Visual Thinking and Icons • Semiotics – the study of signs and symbols • Icon – an image, picture or symbol representing a concept • In computer systems usually less than one inch square (64x64 pixels) • Smaller icons are often integrated with a window border or toolbar • Task dependency • When working on a visual task (e.g., painting program), icons may be useful • When working on a text-based task, it may be better to stay text based • Icons with words (or mouse-overs) are useful

35. Direct Manipulation Interfaces • Icon related guidelines • Represent the object or action in a familiar and recognizable manner • Limit the number of different icons • Make the icon standout from its background • 3-d icons can be visually distracting • Ensure that a single selected icon is clearly visible when surrounded by unselected icons • Make each icon distinctive from every other icon • Ensure harmoniousness of each icon as a member of a family of icons • Design the movement animation (e.g., grayed-out ghost image on a drag) • Detail information • Larger shadowing for a larger file • Color to show the age of a document • Animation to show how much of a file has printed (document icon absorbed progressively into the printer icon) • Explore use of combination of icons • (E.g., drag a document to a printer icon)

36. Direct Manipulation Interfaces • Direct Manipulation Programming • Example: programming a radio to a set of stations by pressing/holding a channel selection button • Phone Services

37. Direct Manipulation Interfaces • Direct Manipulation Programming • Programming in the User Interface • Sufficient computational generality • Conditionals (if, then, else) • Iteration (repeat/while) • Access to appropriate data structures and operators • File structures for directories • Addition, subtraction, etc. • Ease in programming • By specification or by demonstration (Flash animation) • Argument passing • Simplicity in invocation and assignment of arguments • Low risk • High probability of bug free programs • Halt and resume

38. Direct Manipulation Interfaces • Direct Manipulation Programming • Viscosity – the difficulty of making changes to a program • Progressive evaluation – the capacity for execution of partial programs

39. Direct Manipulation Interfaces • 3-Dimensional Interfaces • Some applications are designed as 2-D to be simpler than real-world systems • Constrain movement • Limit interface actions • Ensure visibility of interface objects • Enhanced 3D may be better than 3D • Flying through objects • Multiple simultaneous views of objects • X-ray vision • Shrink/expand objects • Group/ungroup components • Going back in time • Less than successful 3D interfaces • Air-traffic control • Showing altitude by perspective drawing only adds clutter when compared to an overview from directly above http://www.youtube.com/watch?v=Hw-O4zX8qRY

40. Direct Manipulation Interfaces • 4-Dimensional Interfaces • 4-D Anyone? Tesseract http://www.youtube.com/watch?v=bIj0oW-tTF4&feature=fvwrel

41. Direct Manipulation Interfaces • Second Life • Multi-user environment where users interact • Users can choose avatars (fantasy images, desirable characteristics)

42. Direct Manipulation Interfaces • Relationship to: • Personality Theory • Social Psychology • Spatial Cognition • Applicable to business meetings, community discussion groups, political forums? • Blaxxun envision

43. Direct Manipulation Interfaces • 3D Desktops and Workplaces • Microsoft’s Task Gallery • Intel’s Grand Canyon • Xerox PARC’s Information Visualizer • No successful products yet

44. Direct Manipulation Interfaces • Tips for effective 3D interfaces • Use occlusion, shadows, perspective and other 3D techniques carefully • Minimize the number of navigation steps for users to accomplish their tasks • Keep text readable (better rendering, good contrast with background, an no more than 30-degree tilt) • Avoid unnecessary visual clutter, distraction, contrast shifts and reflections • Simplify user movement (keep movements planar, avoid surprises like going through walls) • Organize groups of items in aligned structures to allow rapid visual search • Enable users to construct visual groups to support spatial recall (e.g., placing items in corners)

45. Direct Manipulation Interfaces • Guidelines for inclusion of 3D features • Provide users overviews so they can see the big picture • Allow teleportation (rapid context shifts by selecting destination in an overview) • Offer x-ray vision • Provide history keeping (recording, undoing, replaying, editing) • Permit rich user actions on objects (save, copy, annotate, share, send) • Give users control over explanatory text (pop-up, floating, screen tips) • Offer tools to select, mark and measure • Implement dynamic queries to rapidly filter out unneeded items • Enable landmarks to show themselves even at a distance • Allow multiple coordinated views (users can be in more than one place at a time)

46. Direct Manipulation Interfaces Mahru Humanoid Robot Real-Time Teleoperation http://www.youtube.com/watch?v=TJmQqC1nHTU&feature=fvwrel • Teleoperation • Derived from direct manipulation and process control • Physical processes taking place in a remote location • Clean-up in a nuclear reactor • Need adequate feedback in sufficient time to permit effective decision making • Manufacturing • Medicine (consultation, radiology) • Military operations (drones) • Home automation • Answering machines • Security systems • Energy control • Appliances

47. Direct Manipulation Interfaces http://www.youtube.com/watch?v=aFPQfhUArjI&feature=fvwrel • Design to accommodate teleoperation issues • Slow response times and time delays • Transmission delay (time for command to reach the microscope) • Operation delay (time until the microscope responds) • Incomplete feedback • The microscope can transmit its current position, but operates so slowly it cannot indicate the exact current position • Unanticipated interferences • The slide is accidentally moved by a person at the local site • May be better for the user to specify a destination (rather than a motion) and wait until the action is completed

48. Direct Manipulation Interfaces • Telemedicine • Remote examination • Remote surgery • Telepathology • Magnification • Focus • Illumination • Position

49. Direct Manipulation Interfaces • Virtual and Augmented Reality • “Being in” as opposed to “Looking at” • Architectural applications • Wall-sized image to give perspective • Animation to simulate movement (left to right) • Treadmill to simulate walking toward, walk through doors, stairs • Replace projector with a head-mounted display • Some applications are better when “looked at” • Air-traffic control • To surgeons want to “Be in” the patients body? • Training using virtual reality • Fifth Dimension Technologies www.5DT.com

50. Direct Manipulation Interfaces • Virtual and Augmented Reality • The CAVE (National Center for Supercomputing Applications) • An immersive virtual reality facility designed for the exploration of and interaction with spatially engaging environments. • The stereoscopic capabilities, coupled with its uniquely immersive design, enable scientists and researchers to interact with their data • An atmospheric scientist can actually "climb inside" of a hurricane and visualize its complex and chaotic elements from any angle or visual perspective • A biological researcher, examining a tightly coiled strand of DNA, can virtually "unravel" this strand and manipulate it in an environment that preserves the critical depth information of the data. • Teach a child to cross a street • http://www.youtube.com/watch?v=C5jSSKwBbVM