Project Paradigms 2

Project Paradigms 2 Research & Project Methods SECC604 Professor Julian Newman 08/10/08

To Cover in these Lectures • Develop and Evaluate – “Proof of Concept”? • Experiments and Hypothesis Testing • Surveys • Comparison with Experiments etc • Elementary ideas about sampling and stats • Simulations • Ethnography • Case Studies • Comparison of Paradigms

Simulation (1) • Simulation depends on having • A model of the system to be studied • Appropriate tools for processing that model to see how it behaves • Simulation is particularly useful where we are dealing with large and complex systems, and want to explore behaviour of whole system in response to changes • “Complex systems are counterintuitive” (Forrester)

Simulation (2) • Logic of a Simulation study is similar to that of an Experiment • Investigator may have a hypothesis he/she wants to test • Or may want to investigate the effects of a design or of a particular design decision • Simulation may be used where manipulation of real system is either impossible or too expensive. • Simulations are like experiments in that we can only TEST ideas, we cannot PROVE them.

Simulation (3) • A digital simulation is not the only kind of simulation. • Any situation which is used as a model of another system can be viewed as a simulation. • E.g. In early 1800s Telford modelled the use of cast iron chains to support suspension bridge – a physical model of the bridge.

Simulation (4) • There is always the possibility that the “analogy” breaks down – if a simulation can be checked against reality, this gives us more confidence in the results. • But remember that a law may have a limited scope: e.g. in optics, Snell’s Law does not apply to certain substances such as Felspar; in Computing, it is suggested that Moore’s Law will be broken by Quantum Computing. • Therefore we can never be certain that a model will work outside the parameters with which it has been tested. • In building a digital simulation, we will have made certain assumptions: how well-founded are they? Need to address Validation and Calibration of model.

Ethnography (1) • “Ethnography” originally meant the study of peoples (“ethnos” = “people” – cf “ethnic”) • Ethnographers, also called “Anthropologists”, used to go and live with “primitive” peoples and study their way of life • More recently the same approach has been adopted for studying social groups and organisations in “modern” societies – sometimes also called “participant observer” approach

Ethnography (2) • Ethnographers typically keep very detailed field notes – like a diary – reflecting on what they have observed and experienced. • Ethnographers write up accounts of their field experiences as “thick descriptions”. • Some researchers in Information Systems advocate use of Ethnographic approaches in order to better understand group work before trying to provide computer support for it, or to understand problems with existing computer systems in organisations.

Ethnography (3) • What can happen if we try to automate support for teamwork, without a close understanding of the roles and activities that make up the team? • Big concern about trying to computerise patient records in the National Health Service (CfH/NPfIT) • Many different professional roles in health care team. • Do the computer professionals understand the health care team’s work? Did they even try?

Ethnography (4) Examples • Team at Lancaster University studied Air Traffic Control – spotted problems that would arise in proposed computerisation. Their predictions were proved correct – there were big problems when computerised ATC first introduced. • Ulrike Schutze carried out a one year Ethnographic study of Knowledge Work in an organization – four days a week in firm, wrote up notes every night, one day a week in university discussing with her supervisor.

Ethnography (5) • Ethnographic studies are time consuming. • It has been found difficult to turn the results into usable design recommendations – critics have called Ethnography “navel gazing” • Could be used within an MSc, but would need to be on a part time basis with daily access to appropriate “site”. May raise ethical, data protection, access, IPR and business confidentiality issues.

Case Study (1) • A Case Study is an in-depth study of a particular situation (e.g. a particular organisation) • Various sources of information may be drawn upon – interviews, documents, observation, etc • Approach may be called “idiographic” – it gives an account of the particular, but is hard to generalise from (contrast with “nomothetic”). • May raise access, IPR and business confidentiality issues.

Case Study (2) • Sometimes the term “Case Study” is also used to refer to a non-research-oriented Develop and Test project, oriented towards an individual client organisation. • Sometimes the term “Case Study” is used for a source of real-world Use Cases that can be employed as “test cases” for a new technological development.

Case Study (3) • Case Studies as source of real-world Use Cases: e.g. in the DIECoM project, we used Use Cases from Automotive and Aerospace industries, but aiming at more general lessons about integrated Configuration Management for Hardware and Embedded Software. • DIECoM used traditional workflow. We now want to re-use the DIECoM cases with a different (agent-based) technology – these Case Studies can be seen as “benchmarks”.

Comparing Paradigms (1) • Develop & Test projects versus non-development projects • Develop & Test projects may be motivated by • Immediate Need to provide functionality (practical problem) • Need to understand more general principles (research problem)

Comparing Paradigms (2) • Immediate Need to provide functionality (practical problem) • We say “Develop & Test” but it should really be “Develop & Evaluate” • Even if an immediate need motivates the development, we should apply a scientific approach to investigating the requirements and evaluating the results • So a good evaluation may have a lot in common with a research study

Comparing Paradigms (3) • Develop & Test to understand more general principles (research problem) • Use some kind of standard application as a benchmark • Repeat the development using different technologies • Then compare the process and/or results • This has some things in common with an experimental paradigm

Comparing Non-Development Project Paradigms • Experiments, Surveys, Simulations • These allow in principle for “replication” – another investigator could follow same procedure and see if they get same results • Usability Labs, Ethnography, Case Studies • These do not in principle allow for “replication” – there is no sufficiently defined procedure to allow another investigator to repeat the “same” study or a systematic variation of it • One might make an exception in the use of Case Studies as Benchmarks – they give a development project some of the characteristics of experimental replicability • Therefore Experiments, Surveys and Simulations appear more “scientific”; however they may miss important insights that can be gained by the other less well controlled methods • Replicability may also depend on experimenter’s skill

Project Paradigms 2