The Analyst’s Tool

1. The Analyst’s Tool and its use in the construction of Groupware systems Alex Feinman afeinman@cs.brandeis.edu

2. Developing Groupware • Developing Groupware systems is HARD • Users don’t know what they need • Developers don’t know what the users need • Solutions are of mixed usage • Forcing the users to use what you come up with • Design Guidelines • E.g., GUI designs; Patterns • Iterative Design • Design -> Implement -> Evaluate -> Redesign …

3. Iterative Design • Slow, Expensive • Requires many cycles for useful software • Initial design may be very far from acceptable product • Often users are not patient enough for a polished product to be produced • Same problems are solved over and over again by different developers • Exponentially more difficult for multi-user system (# subgroups = 2n)

4. Iterative Design: Evaluation • Much work has been done on Design (e.g., GOMS) and Implementation (e.g., ADAPTIVE) • Less work on Evaluation • More precise evaluation should allow for fewer cycles, leading to faster development • Multi-user evaluation has its own extra complications

5. Analyst’s Tool • Part of a larger system for implementing and analyzing Groupware Analyst’s Tool More Groupware Systems VesselWorld SAGE (replay tool) Basic Groupware Tools (chat room, shared whiteboard, etc.) ADAPTIVE

6. Methodology • Implement a base groupware system, providing only basic tools such as a chat room and shared whiteboard • Collect significant amount of data of real workgroups doing their task using the base system • Analyze data using analyst’s tool to isolate and evaluate coordination issues • Implement Coordination Representations to solve recurring problems of coordination

7. Guidelines for Building Coordinating Representations • Library of patterns in coordinating representations is under construction • list, checklist, set, table, flowchart • media attributes: Clark’s list [reviewable, editable, synchronous, etc.]; verification needed, approval needed, vetoable; voluntary vs. automatic updates; etc. • interface design issues: screen real estate, effort to create/modify/remove items, • Guidelines for matching coordination problems to attributes also under construction • e.g., if a known of actions needs to be performed in order, a checklist may be indicated • First, need gather and analyze data to determine which coordinating representations are needed

8. Gathering Data • ADAPTIVE automatically collects data • Data collected consists of: • Chat transcripts: • GUI actions performed by users • mouse clicks, window movement, keystrokes, etc. • Domain-specific actions of users • in VesselWorld, lifting and moving waste, moving barges, navigating the harbor, etc. crane2: let’s get swaste1 crane1: no way dude. We need to wait for teh tug crane2: tug, isn’t swaste1 no eq? crane2: no, wait, it’s unknown tug1: haven’t id’d it yet crane2: so wait crane1: we could do the xl int he corner

9. Analyzing the Data • Primarily analyzing the chat transcript, supported by investigation of domain actions • Chat can be analyzed using augmentation of Conversation Analysis techniques • Four basic areas of interest • Segmenting • Turn construction • Turn-taking • Group dynamics

10. Segmenting • Segment tool allows analyst to easily delineate interesting sections of data • Allows alignment of corresponding sections of similar segments and extract the shared structure • Allows construction of segment hierarchy, from which a hierarchy of actions can be inferred • Once a case-base of possible user actions has been constructed, user choices in turn-construction are made more visible

11. Segmenting Tool Segment Comparison View Segment A: jerry segment Segment B: are you sure that’s the net crane2: so I says to him, jerry, I says, I don’t want to go crane1: quite and submit something tug1: waste at 300/125 needs net crane1: what size? crane2: and he says, shut up and be quick about it. crane1: [tug1] What size is the waste at 300, 125? tug1: waste at 320/98 needs nothing tug1: net crane1: [tug1] not equipment, what size is it? crane2: doesn’t anyone want to hear my storiy? tug1: small waste at 300/125 crane1: whatst hew aste at 300 190? crane1: oops 190 300 crane2: ln@300,190!l crane1: thanks tug1: Are you sure that’s net? crane2: I thought so tug1: I had ‘no equipment’. Let me check before you get it. crane1: ok. FRAME: WASTE INFORMATION QUERY ------------------------------------ SLOT 1: NAME: Request for information COMPLICATIONS: no response (busy, ignoring?) (A:1b) self-correction (B:1b) correction after next (no examples) STRUCTURES: none recorded yet ------------------------------------ SLOT 2: NAME: Information returned COMPLICATIONS: mismatch of CG between users (B:2c-2e) wrong information returned (A:2a-2c) STRUCTURES: jargon (B:2a) * 1a * 1b 2a * 2b 1a 2c * 2d 1b 2e * * 2a 2b * 2c

12. Turn construction • Analyst can tag individual utterances as desired to manually extract patterns • Two sets of tags • Domain-independent list of tags, generally dealing with coordination issues (e.g., Commitment, Reference Error) • Domain-specific list of tags (e.g., Waste Dropped, Joint Lift) • Tag lists are customizable by analyst

13. Turn-taking • Quantitative analysis tools allow examination of trends in turn-taking • Turn-taking clusters may point to subgroup formation • Patterns in turn-taking may indicate negotiation of roles

14. Group Dynamics • Structured notepad allows analyst to record impressions of group dynamics • Overview/timeline tool presents impressions in situ, allowing comparison of dynamics across problem-solving sessions and groups.

15. Statistical Analysis Tool • Allows rapid investigation into statistical trends in the data. • Currently under development by Michael Alex

16. Integration of Tools • Tools not meant to be used in isolation • e.g., segmenting and comparing segments allows easier extraction of case-base of (conversational) actions • Tool data is persistent and shareable, allowing collaboration by multiple analysts either concurrently or in sequence • Tool could perform automatic comparison of analysis by multiple reviewers, strengthening confidence in results