Interaction Devices

Interaction Devices Human Computer Interaction CIS 6930/4930 Section 4188/4186

Interaction Performance • 60s vs. Today • Performance • Hz -> GHz • Memory • k -> GB • Storage • k -> TB • Input • punch cards -> • Keyboards, Pens, tablets, mobile phones, mice, digital cameras, web cams • Output • 10 character/sec • Megapixel displays, color laser, surround sound, force feedback, VR • Substantial bandwidth increase!

Interaction Performance • Future? • Gestural input • Two-handed input • 3D I/O • Others: voice, wearable, whole body, eye trackers, data gloves, haptics, force feedback • Engineering research! • Entire companies created around one single technology • Current trend: • Multimodal (using car navigation via buttons or voice) • Helps disabled (esp. those w/ different levels of disability)

Keyboard and Keypads • QWERTY keyboards been around for a long time • (1870s – Christopher Sholes) • Cons: Not easy to learn • Pros: Familiarity • Stats: • Beginners: 1 keystroke per sec • Average office worker: 5 keystrokes (50 wpm) • Experts: 15 keystrokes per sec (150 wpm) • Is it possible to do better? Suggestions?

Keyboard and Keypads • Look at the piano for possible inspiration • Court reporter keyboards (one keypress = multiple letters or a word) • 300 wpm, requires extensive training and use • Keyboard properties that matter • Size • large - imposing for novices, appears more complex • mobile devices • Adjustable • Reduces RSI, better performance and comfort • Mobile phone keyboards, blackberry devices, etc.

Keyboard Layouts • QWERTY • Frequently used pairs far apart • Fewer typewriter jams • Electronic approaches don’t jam.. why use it? • DVOARK (1920s) • 150 wpm->200 wpm • Reducing errors • Takes about one week to switch • Stops most from trying • ABCDE – style • Easier for non-typists • Studies show no improvement vs. QWERTY • Number pads • What’s in the top row? • Look at phones (slight faster), then look at calculators, keypads • Those for disabled • Split keyboards • KeyBowl’s orbiTouch (screenshot) • Eyetrackers, mice • Dasher - 2d motion with word prediction

Keys • Current keyboards have been extensively tested • Size • Shape • Required force • Spacing • Speed vs. error rates for majority of users • Distinctive click gives audio feedback • Why membrane keyboards are slow (Atari 400?) • Environment hazards might necessitate • Usually speed is not a factor

Keys Guidelines • Special keys should be denoted • State keys (such as caps, etc.) should have easily noted states • Special curves or dots for home keys for touch typists • Inverted T Cursor movement keys are important (though cross is easier for novices) • Auto-repeat feature • Improves performance, but only if repeat is customizable (motor impaired, young, old) • Two thinking points: • Why are home keys fastest to type? • Why are certain keys larger? (Enter, Shift, Space bar) • This is called Fitt’s Law

Keypads for small devices • PDAs, Cellphones, Game consoles • Fold out keyboards • Virtual keyboard • Cloth keyboards (ElekSen) • Haptic feedback? • Mobile phones • Combine static keys with dynamic soft keys • Multi-tap a key to get to a character • Study: Predictive techniques greatly improve performance • Ex. LetterWise = 20 wpm vs 15 wpm multitap • Draw keyboard on screen and tap w/ pen • Speed: 20 to 30 wpm (Sears ’93) • Handwriting recognition (still hard) • Subset: Graffiti2 (uses unistrokes)

Pointing Devices • Direct manipulation needs some pointing device • Factors: • Size of device • Accuracy • Dimensionality • Interaction Tasks: • Select – menu selection, from a list • Position – 1D, 2D, 3D (ex. paint) • Orientation – Control orientation or provide direct 3D orientation input • Path – Multiple poses are recorded • ex. to draw a line • Quantify – control widgets that affect variables • Text – move text • Faster w/ less error than keyboard • Two types (Box 9.1) • Direct control – device is on the screen surface (touchscreen, stylus) • Indirect control – mouse, trackball, joystick, touchpad

Direct-control pointing • First device – lightpen • Point to a place on screen and press a button • Pros: • Easy to understand and use • Very fast for some operations (e.g. drawing) • Cons: • Hand gets tired fast! • Hand and pen blocks view of screen • Fragile • Evolved into the touchscreen • Pros: Very robust, no moving parts • Cons: Depending on app, accuracy could be an issue • 1600x1600 res with acoustic wave • Must be careful about software design for selection (land-on strategy). • If you don’t show a cursor of where you are selecting, users get confused • User confidence is improved with a good lift-off strategy

Direct-control pointing • Primarily for novice users or large user base • Case study: Disney World • Need to consider those who are: disabled, illiterate, hard of hearing, errors in usage (two touch points), etc.

Indirect-Control Pointing • Pros: • Reduces hand-fatigue • Reduces obscuration problems • Cons: • Increases cognitive load • Spatial ability comes more into play • Mouse • Pros: • Familiarity • Wide availability • Low cost • Easy to use • Accurate • Cons: • Time to grab mouse • Desk space • Encumbrance (wire), dirt • Long motions aren’t easy or obvious (pick up and replace) • Consider, weight, size, style, # of buttons, force feedback

Indirect-Control Pointing • Trackball • Pros: • Small physical footprint • Good for kiosks • Joystick • Easy to use, lots of buttons • Good for tracking (guide or follow an on screen object) • Does it map well to your app? • Touchpoint • Pressure-sensitive ‘nubbin’ on laptops • Keep fingers on the home position

Indirect-Control Pointing • Touchpad • Laptop mouse device • Lack of moving parts, and low profile • Accuracy, esp. those w/ motor disabilities • Graphics Tablet • Screen shot • comfort • good for cad, artists • Limited data entry

Comparing pointing devices • Direct pointing • Study: Faster but less accurate than indirect (Haller ’84) • Lots of studies confirm mouse is best for most tasks for speed and accuracy • Trackpoint < Trackballs & Touchpads < Mouse • Short distances – cursor keys are better • Disabled prefer joysticks and trackballs • If force application is a problem, then touch sensitive is preferred • Vision impaired have problems with most pointing devices • Use multimodal approach or customizable cursors • Read Vanderheiden ’04 for a case study • Designers should smooth out trajectories • Large targets reduce time and frustration

Example • Five fastest places to click on for a right-handed user?

Example • What affects time?

Fitts’s Law • Paul Fitts (1954) developed a model of human hand movement • Used to predict time to point at an object • What are the factors to determine the time to point to an object? • D – distance to target • W – size of target • Just from your own experience, is this function linear? • No, since if Target A is D distance and Target B is 2D distance, it doesn’t take twice as long • What about target size? Not linear there either • MT = a + b log2(D/W + 1) • a = time to start/stop in seconds (empirically measured per device) • b = inherent speed of the device (empirically measured per device) • Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm • Ans: 300 + 200 log2(14/2 + 1) = 900 ms

Fitts’s Law • MT = a + b log2(D/W + 1) • a = time to start/stop in seconds (empirically measured per device) • b = inherent speed of the device (empirically measured per device) • Ex. a = 300 ms, b = 200 ms/bit, D = 14 cm, W = 2 cm • Ans: 300 + 200 log2(14/2 + 1) = 900 ms • Question: If I wanted to half the pointing time (on average), how much do I change the size? • Proven to provide good timings for most age groups • Newer versions taken into account • Direction (we are faster horizontally than vertically) • Device weight • Target shape • Arm position (resting or midair) • 2D and 3D (Zhai ’96)

Very Successfully Studied • Applies to • Feet, eye gaze, head mounted sights • Many types of input devices • Physical environments (underwater!) • User populations (even retarded and drugged) • Drag & Drop and Point & Click • Limitations • Software accelerated pointer motion • Training • Trajectory Tasks (Accot-Zhai Steering Law) • Decision Making (Hick’s Law) • Results (what does it say about) • Buttons and widget size? • Edges? • Popup vs. pull-down menus • Pie vs. Linear menus

Precision Pointing Movement Time • Study: Sears and Shneiderman ’91 • Broke down task into gross and fine components for small targets • PPMT = a + b log2(D/W+1) + c log2(d/W) • c – speed for short distance movement • d – minor distance • Notice how the overall time changes with a smaller target. • Other factors • Age (Pg. 369) • Research: How can we design devices that produce smaller constants for the predictive equation • Two handed • Zooming

Novel Devices • Themes: • Make device more diverse • Users • Task • Improve match between task and device • Improve affordance • Refine input • Feedback strategies • Foot controls • Already used in music where hands might be busy • Cars • Foot mouse was twice as slow as hand mouse • Could specify ‘modes’

Novel Devices • Eye-tracking • Accuracy 1-2 degrees • selections are by constant stare for 200-600 ms • How do you distinguish w/ a selection and a gaze? • Combine w/ manual input • Multiple degree of freedom devices • Logitech Spaceball and SpaceMouse • Ascension Bird • Polhemus Liberty and IsoTrack

Novel Devices • Boom Chameleon • Pros: Natural, good spatial understanding • Cons: limited applications, hard to interact (very passive) • DataGlove • Pinch glove • Gesture recognition • American Sign Language, musical director • Pros: Natural • Cons: Size, hygiene, accuracy, durability

Novel Devices • Haptic Feedback • Why is resistance useful? • SensAble Technology’s Phantom • Cons: limited applications • Sound and vibration are easier and can be a good approximation • Rumble pack • Two-Handed input • Different hands have different precision • Non-dominant hand selects fill, the other selects objects • Ubiquitous Computing and Tangible User Interface • Active Badges allows you to move about the house w/ your profile • Which sensors could you use? • Elderly, disabled • Research: Smart House • Myron Kruger – novel user participation in art (Lots of exhibit art at siggraph)

Novel Devices • Paper/Whiteboards • Video capture of annotations • Record notes (special tracked pens Logitech digital pen) • Handheld Devices • PDA • Universal remote • Help disabled • Read LCD screens • Rooms in building • Maps • Interesting body-context-sensitive. • Ex. hold PDA by ear = phone call answer.

Novel Devices • Miscellaneous • Shapetape – reports 3D shape. • Tracks limbs • Engineer for specific app (like a gun trigger connected to serial port) • Pros: good affordance • Cons: Limited general use, time

Speech and Auditory Interfaces • There’s the dream • Then there’s reality • Practical apps don’t really require freeform discussions with a computer • Goals: • Low cognitive load • Low error rates • Smaller goals: • Speech Store and Forward (voice mail) • Speech Generation • Currently not too bad, low cost, available

Speech and Auditory Interfaces • Bandwidth is much lower than visual displays • Ephemeral nature of speech (tone, etc.) • Difficulty in parsing/searching (Box 9.2) • Types • Discrete-word recognition • Continuous speech • Voice information • Speech generation • Non-speech auditory • If you want to do research here, lots of research in the audio, audio psychology, and DSP field you should understand

Discrete-Word Recognition • Individual words spoken by a specific person • Command and control • 90-98% for 100-10000 word vocabularies • Training • Speaker speaks the vocabulary • Speaker-independent • Still requires • Low noise operating environment • Microphones • Vocabulary choice • Clear voice (language disabled are hampered, stressed) • Reduce most questions to very distinct answers (yes/no)

Discrete-Word Recognition • Helps: • Disabled • Elderly • Cognitive challenged • User is visually distracted • Mobility or space restrictions • Apps: • Telephone-based info • Study: much slower for cursor movement than mouse or keyboard (Christian ’00) • Study: choosing actions (such as drawing actions) improved performance by 21% (Pausch ’91) and word processing (Karl ’93) • However acoustic memory requires high cognitive load (> than hand/eye) • Toys are successful (dolls, robots). Accuracy isn’t as important • Feedback is difficult

Continuous Speech Recognition • Dictation • Error rates and error repair are still poor • Higher cognitive load, could lower overall quality • Why is it hard? • Recognize boundaries (normal speech blurs them) • Context sensitivity • “How to wreck a nice beach” • Much training • Specialized vocabularies (like medical or legal) • Apps: • Dictate reports, notes, letters • Communication skills practice (virtual patient) • Automatic retrieval/transcription of audio content (like radio, CC) • Security/user ID

Voice Information Systems • Use human voice as a source of info • Apps: • Tourist info • Museum audio tours • Voice menus (Interactive Voice Response IVR systems) • Use speech recognition to also cut through menus • If menus are too long, users get frustrated • Cheaper than hiring 24 hr/day reps • Voice mail systems • Interface isn’t the best • Get email in your car • Also helps with non-tech savvy like the elderly • Potentially aides with • Learning (engage more senses) • Cognitive load (hypothesize each sense has a limited ‘bandwidth’) • Think ER, or fighter jets

Speech Generation • Play back speech (games) • Combine text (navigation systems) • Careful evaluation! • Speech isn’t always great • Door is ajar – now just a tone • Use flash • Supermarket scanners • Often times a simple tone is better • Why? Cognitive load • Thus cockpits and control rooms need speech • Competes w/ human-human communication

Speech Generation • Ex: Text-to-Speech (TTS) • Latest TTS uses multiple syllabi to make generated speech sound better • Robotic speech could be desirable to get attention • All depends on app • Thus don’t assume one way is the best, you should user test • Apps: TTS for blind, JAWS • Web-based voice apps: VoiceXML and SALT (tagged web pages). • Good for disabled, and also for mobile devices • Use if • Message is short • Requires dynamic responses • Events in time • Good when visual displays aren’t that useful. When? • Bad lighting, vibrations (say liftoff)

Non-speech Auditory Interface • Audio tones that provide information • Major Research Area • Sonification – converting information into audio • Audiolization • Auditory Interfaces • Browsers produced a click when you clicked on a link • Increases confidence • Can do tasks without visual cognitive load • Helps figure out when things are wrong • Greatly helps visually impaired

Non-speech Auditory Interface • Terms: • Auditory icons – familiar sounds (record real world sound and play it in your app) • Earcons – new learned sounds (door ajar) • Role in video games is huge • Emotions, Tension, set mood • To create 3D sound • Need to do more than stereo • Take into account Head-related transfer function (HRTF) • Ear and head shape • New musical instruments • Theremin • New ways to arrange music

Displays • Primary Source of feedback • Properties: • Physical Dimension • Resolution • Color Depth and correctness • Brightness, contrast, glare • Power • Refresh rate • Cost • Reliability • # of users

Display Technology • Monochrome displays (single color) • Low cost • Greater intensity range (medical) • Color • Raster Scan CRT • LCD – thin, bright • Plasma – very bright, thin • LED – large public displays • Electronic Ink – new product w/ tiny capsules of negative black particles and positive white • Braille – refreshable cells with dots that rise up

Large Displays • Wall displays • Informational • Control rooms, military, flight control rooms, emergency response • Provides • System overview • Increases situational awareness • Effective team review • Old: Array of CRTs • Interactive • Require new interaction methods (freehand sketch, PDAs) • Local and remote collaboration • Art, engineering

Large Displays • Multiple Desktop Displays • Multiple CRTs or Flat panels for large desktops • Cheap • Familiar • Spatial divide up tasks • Comparison tasks are easier • Too much info? • HMD • Eventually -> Every surface a pixel

Mobile device displays • Applications • Personal • Reprogrammable picture frames • Digital family portrait (GaTech) • Business • PDAs, cellphones • Medical • Monitor patients • Research: Modality Translation Services (Trace Center – University of Wisconsin) • As you move about it auto converts data, info, etc. for you

Mobile device displays • Actions on mobile devices • Monitor information and alert (calendar) • Gather then spread out information (phone) • Participate in groups and relate to individual (networked devices) • Locate services and identify objects (GPS car system) • Capture and then share info (phone)

Mobile device displays • Guidelines for design • Bergman ’00, Weiss, ’02 • Industry led research and design case studies (Lindholm ’03) • Typically short in time usage (except handheld games) • Optimize for repetitive tasks (rank functions by frequency) • Research: new ways to organize large amounts of info on a small screen • Study: Rapid Serial Visual Presentation (RSVP) presents text at a constant speed (33% improvement Oquist ’03) • Searching and web browsing still very poor performance • Promising: Hierarchical representation (show full document and allow user to select where to zoom into)

Animation, Image, and Video • Content quality has also greatly increased • 3D rendering is near life-like • Digital Photography is common • Scanned documents • Video compression • Multimedia considerations for the disabled • Printers • 3D Printers create custom objects from 3D models

Interaction Devices