550 likes | 904 Views
CSCW Evaluation Techniques . Presented by: Christopher Edwards. Overview of Presentation. Evaluation Techniques Understanding Ethnography Using Ethnography in CSCW Understanding Ethnomethodology Ethnomethodology and CSCW Technomethodology Conclusion. Evaluation Overview.
E N D
CSCW Evaluation Techniques Presented by: Christopher Edwards
Overview of Presentation • Evaluation Techniques • Understanding Ethnography • Using Ethnography in CSCW • Understanding Ethnomethodology • Ethnomethodology and CSCW • Technomethodology • Conclusion
Evaluation Overview • Olson and Olson. • What are we Evaluating? • Evaluations Techniques… • Internal/External Validity • Conclusion
Characteristics of Groups • Individuals Differ in: • Skills • Ability • Knowledge • Personalities • Motivations • Agendas
Characteristics of Organizations • System comprised of people and technology • Social Technology • Physical Technology • Comprised of multiple actors • Dependency on Communication • Information Processing Entities
Characteristics of Task • Tasks involve different types of material • Physical, Digital or Ethereal • Ease or Difficulty of Task • Differ on Core Activity • Subtasks – Tightly Coupled/Loosely coupled
Characteristics of Environment • Physical Environment • Distance between Group members • CSCW technologies designed to overcome • Contextual Time • When in the day the interaction occurs • Effects on Distant Group member
Characteristics of Technology • Increasingly Varied • Technologies to Support Conversation • Auditory/Visual • Back channels/gestures • Technologies to Support Shared Work • Objects that support work • Fit of Tool to Material
Process • Technology Deployment • How and Why • Process Analysis • Why outcomes were affected • Progress of Task • Communication process • Examined through time scales
Outcomes • The initial outcome of using technology • Quality of work • Measure outcomes at every level • Group Outcome • Organizational Productivity
Conceptual Framework for CSCW Studies Group Organization Process Outcomes Task Environment Technology
The Survey • Set of questions • Fixed Alternatives • Statistically Analyzed • Wording of questions problematic
Interview • Structure of interview • Formal and structured • Unstructured • Analysis can be complicated
Experimental • Controlled Setting • Specific Task • Conditions • Assignment of Participants • Useful for making inferences about causality
Case Study • Examines a single or small number of cases • Exploratory research
Ethnography • Method adopted from Anthropology • Describing Culture • Used originally to describe other cultures • Misunderstood method
Many other Methods • Diaries • Analytic Field Studies • Quasi Experimental • Longitudinal Studies • Historical Studies
Internal and External Validity HIGH Laboratory Experiments Field Experiments Level of Internal validity Surveys Ethnographies LOW Level of External Validity HIGH
Conclusion to the Overview • Different factors influence use and evaluation of CSCW software • Framework of CSCW studies • Evaluation Techniques • Validity of Techniques
Short Break • Reconvene in 3 minutes • *Upcoming – Understanding Ethnography • Using Ethnography • Ethnography and CSCW
Ethnography • Understanding Ethnography • Sociology Adoption • Using Ethnography • CSCW Ethnography in Design (Hughes)
Understanding Ethnography • Ethnography is loosely applied to qualitative research • Home is originally from Anthropology • Aim to describe cultural interpretation
Understanding Ethnography • Understanding culture “from an insiders point view” • Three sources of data • Participant Observation • Interviews • Collection of representative artifacts
Sociology Adoption • Originally used to study distant cultures • Chicago School of Sociology • Studies focused on exploration of groups in urban settings • Cultural comparisons in USA • Family of Ethnographic Techniques
Class Participation Time • In Pairs (Saul and myself included) • Everyone gets a Handout • For a total of 5 Minutes (2.5 Minutes each) • Each member of the pair (one at a time) asks the other questions from the sheet
Debrief of Class participation • What answers were given…. • Obviously not a long term ethnography study • Depending on your relationship to this lab – differing perspectives • Understanding Grouplab culture (to some extent)
Ethnography and CSCW • Prominence of Ethnography in CSCW • Insufficient attention to social context • New problems for design of collaborative character of work and activities • Ethnography and system design • Problem of scale • Pressure of time • Role of the ethnographer
Concurrent Ethnography • Design is influenced by on-going ethnographic study • Sequenced process Ethnographic Study Systems Development Debriefing Meetings System Prototype
Concurrent Ethnography in action • London Air Traffic Control Centre • Four week Ethnography • Each stage of fieldwork was intended to target designers issues • Small research team • What ethnography provided
Quick and Dirty Ethnography • Brief Ethnographic Studies • Duration relative to the size of the task • Selecting aspects of work setting of importance to design Outline of project Meetings Short Focus Studies Debriefing Meetings Scoping Document
Quick and Dirty Ethnography in Action • Ethnographic investigation of software engineers • Challenges of Large scale setting • Working in Industrialized Environments • Acceptance into the setting (*Key to Ethnographic research)
Evaluative Ethnography • Ethnography used to verify formulated design decisions Initial outline Design or Specification Short Ethnographic study Debriefing Meetings Amended Design Or Specification
Evaluative Ethnography in action • Fieldwork in Building Society • Using research for IT developments • Routine of work • Finding what customers wanted • Outlined limitations of model that had been proposed
Re-examination of previous studies • Previous studies are re-examined to inform design • Ethnography used for many decades • Many studies related to work and occupation • Can be informative
Re-examination in action • Inform preliminary design of Shared Object Service • Using previous Ethnographic studies on: • Social work, police work and invoice processing in a multi-site fast food company • What common service should support
Summary of Ethnography • Understanding Ethnography • Ethnography and CSCW • Uses of Ethnography • Concurrent • Quick and Dirty • Evaluative • Re-examination
Big Break Time 5 Minutes • Reconvene in 5 Minutes • Upcoming – Ethnomethodology • Understanding Ethnomethodology • Ethnomethodology in CSCW
Ethnomethodology • Understanding Ethnomethodology • Confusing Ethnography and Ethnomethodology • Ethnomethodology in system design • Incorporation of Sociology and Computer Science? ‘Technomethodology’
Understanding Ethnomethodology • Ethnomethodology literally means “People’s Methods” • A Shift from ‘other’ Sociological Methods • Social Life is potentially Chaotic • Social Actors • Members methods for making sense
Understanding Ethnomethodology • Garfinkel “Documentary Method” • Example of Documentary Method • Garfinkel “Indexicality” • Disrupt Technique • Example in class
Understanding Ethnomethodology • We can observe other members methods of construction • Development of Conversation Analysis
Confusing Ethnography and Ethnomethodology • Ethnography is a form of investigative fieldwork • Ethnography focuses on the “Member’s Point of View” • Ethnomethodology is a specific analytical technique
Confusing Ethnography and Ethnomethodology • Confusion arises because: • Ethnomethodologist is likely to use ethnographic techniques • ‘Analytic mentality’-selection of phenomena and topics for investigation
Ethnomethodology in HCI and CSCW • Observations of work activities and interactions help design process • Understanding temporal organization of activities and interactions and implications to design
Learning from Ethnomethodologists • Division of Labour • Field Observation conducted by ethnomethodologists • Act as proxy for end users • Hand off requirements to computer science people
Ethnomethodology for Critique and Design • Ethnomethodology has provided: • Critique of the design • Failure to support the work • Technology doesn’t allow people to engage in their work • Outlines organization of work and communication in the real world
Two Paradoxes • Paradox of system design • Large scale activity • Paradox of technomethodology • Transformational nature of technology • Analysis of practice not invention
Technomethodology • Develop a stance in which ethnomethodology and computer science play equally significant roles • Foundational relationships
Technomethodology • Trying to exploit generalizations from ethnomethodology • Abstractions from both disciplines • Means by which such working practices arise • Dialogical interfaces