CMT 3210, Week 9

1. CMT 3210, Week 9 Interaction As a Topic in Its Own Right

2. Topics • Paradigms for interaction • Direct manipulation • Visual and spatial representations, visual thinking • Object action interface model (OAI) • Conversation based interaction (command line style) • Speech acts • Conversation for action • Structure of every day conversations

3. Direct manipulation You act in the computer’s world Act on work-domain objects Windows, Mac Affordances: pressability, dragability, turnability, etc Paradigms for Interaction

4. Paradigms for Interaction • “Conversation-based” interfaces • You ask the computer to act on your behalf • Actions expressed in a command language • “Command line” interfaces (Unix, DOS) • Syntax is important bob12> ls letter.doc letter-old.doc paper.dvi paper.log paper.ps paper.tex bob12> cp paper.ps backup.ps bob12> lpr paper.ps

5. Characteristics of Direct Manipulation • Continuous representation of objects of interest • Physical actions or labelled buttons instead on complex syntax • Rapid incremental reversible operations, whose impact is immediately visible • WYSIWYG principle: What you see is what you get. (1990)

6. Visual & Spatial Representations • Spreadsheets (Visicalc, Lotus notes, Excel) • (Video-) Games (PONG, Gameboy,…) • Computer-aided Design (CAD) • Computer-aided Manufacturing (CAM) • Office automation (Word, Macintosh programs, Windows) • Information from databases.

7. Advantages of Direct Manipulation • Virtuality: • Representation of reality that can be manipulated • Transparency • User applies intellect to the task at hand, the tool itself seems to disappear into the background • Problem solving and learning by using virtual/physical objects • Retention and manipulation of physical, spatial or visual objects is easier

8. Problems • Visual representations can be space consuming • Consequences: scrolling or deeper layers • Users have to learn the meaning of visual representations e.g. Icons • Using a mouse can be slower than typing • Visual representations can be ambiguous and misleading • Visual representations can be inappropriate for certain type of information. • Clarity: “Stop” is clearer than any sign • Amount and detail of Information required: text

9. Structure of Visual Objects, Particularly Icons • Lexical qualities • Pixel shape, colour, brightness, blinking • Syntactics • Appearance and movement: lines, patterns, modular parts, size, shape • Semantics • Type of objects represented: concrete vs. Abstract, part vs. Whole

10. Structure of Visual Objects, Particularly Icons • Pragmatics • Overall legibility, utility, identifiability, memorability, appeal • Dynamics • Receptivity to clicks: Highlightening, dragging, combining, etc.

11. Universe Intention atoms steps Actions Objects Task Object Action Interface Model

12. Object Action Interface Model Metaphor Plan pixels clicks Actions Objects Interface

13. Stages of Development (Piaget) • Sensorimotor (birth to 2 years) • Touching, grabbing, smelling, tasting,… • Pre-operational (2 to 7 years) • Differentiation of shapes, colours, texture, etc • Concrete operational (7 to 11 years) • Manipulation of physical objects • Formal operations (11+ years ) • Symbol manipulation

14. Conversation Based Interaction • User asks the computer to do something on his/her behalf • Syntax is important • Can we learn something for these interfaces from language and interaction theories?

15. Language and Interaction • Linguists understand language on three levels: • Syntax • the surface structure of a communication • Semantics • what it means • Pragmatics • what to do about it

16. Language and Interaction, Example 1 • In the bar: “Do you have any peanuts?” • Syntax: Sentence-Question, do-auxiliary verb, have-verb, peanuts-noun (plural), etc. • Semantics: Somebody asks for peanuts. • Pragmatics: Give him/her peanuts, reject request, ignore request, divert conversation,…

17. Language and Interaction Example 2: • From the compiler: “error 87 at line 36.” • Syntax: incomplete sentence, error-noun, 87-number, at-preposition, line-noun, 36-number. • Semantics: the compiler detected an error, while compiling the source code at line 36. (That does not necessarily mean that the mistake in the source code is at line 36.) • Pragmatics: go back to source code and check from line 36 backwards.

18. Language and Action • The “language/action perspective”: Speech Acts • Language is used to do things, change the state of the world, etc. (Not just to describe) • Utterances are designed to carry out "speech acts" • Speech acts used as part of dialogue between people • Types of speech acts categorise the different functions of utterances

19. Speech Acts…. • Assertives • commit the speaker to something being the case • Directives • attempt to get the listener to do something • Commissives • commit the speaker to some course of action • Expressives • express a psychological state • Declarations • bring about a state of affairs

20. A: two pints of lager please B: coming right up A: thanks A: two pints of lager please B: the lager's off A: I'll have bitter then B: coming right up A: thanks Speech Acts and Dialogue A: two pints of lager please B: would you like bitter instead A: don't think I'll bother B: suit yourself

21. Conversations for Action A: Declare A: Request B: Promise B: Assert A: Declare 1 2 3 4 5 A: Renege B: Counter A: Counter A: Reject 7 6 8 9

22. Potential Applications • Structure of human-computer dialogues • Can we capture the range of human-computer interactions in a similar way? • Categorisations of actions? Or task sequences? • Computer support for human-human dialogues • Specialised communication tools? • Knowing about the structure of conversations can allow special kinds of support to be provided • Project management tools

23. Questions: • Similarities and differences between human-human interaction and human-computer interaction. • Similarities: messages going back and forth • Differences: context sensitivity, non-verbal communication, humor, ‘repair actions’ in case of misunderstanings, metacommunication, etc. • Are humans interacting with a computer or are they interacting with the organisation behind the computer (e.g. cash machines)?

24. Properties of Language • Speech acts considered too restrictive • Study the way language used in ordinary conversation: • Openings, closings, adjacency pairs

25. Openings and Closings: Example A: Do you want another pint? B: what's the time? A: ten to ten B: OK, yes thanks Opening Opening Closing Closing

26. bob12> cp paper.ps temp Disc full bob12> rm paper-old Removed paper-old bob12> cp paper.ps temp copied paper.ps bob12> Openings and Closings Opening ? Opening Closing Opening Closing

27. Action-response pairs First part proposes some joint task Second part may “take up” task of the first Different forms of take-up A: who is speaking? B: it’s Bob Fields C: is he going to the US? D: well, I heard that …. E: who is calling? F: can I leave a message? Adjacancy Pairs

28. Design Applications? • Structure of human-computer conversations • match openings and closings or • categorise responses in dialogue using adjacency pairs?

29. Summary • Paradigms for interaction: • direct manipulation • conversation-based / command-line • Visual representation and OAI models • Language/action perspective • Conversation analysis • openings, closings and adjacency pairs

30. Further reading E. Hutchins, J. Hollan & D. Norman (1986) Direct Manipulation Interfaces. In Norman & Draper (eds, 1986) User Centred System Design. LEA. T. Winograd & F. Flores (1987) Understanding computers and cognition: a new foundation for design. Addison-Wesley. H. Clark (1997) Using Language. CUP.