Research Process: Steps, Methods, and Ethics

1. IS 4800 Empirical Research Methods for Information Science Class Notes April 6, 2012

2. Steps in the research process • Identify a phenomenon of interest • Iterate: • Investigate current state of knowledge (lit. review ?) • Narrow down your interest to a research question or hypothesis • Identify research method to employ (survey, experiment, ethnography, case study)

3. Steps in the research process (cont.) • Operationalize research question or hypothesis • Define the source of your data • Sample population and recruitment method (if relevant) • And/or the location/activities to be observed • Define variables and/or data collection methods and instruments • Identify how the analysis will be carried out ----YOU NOW HAVE A RESEARCH PROPOSAL ------

4. Steps in the research process (cont.) • Carry out your observations • Analyze your data • Draw conclusions, write up the results

5. Qualitative Data and Collection Methods • Direct observation • Participant observation • In-depth interviews • Focus groups • “Artifacts” – usually text or Databases

6. Direct Observation • May be in person or use audio or videotape, observe through a 1-way mirror • Unlike participant observation, often focused on specific events (how many, how often, by whom, observe patterns – for example, interruptions at a meeting)

7. What to observe • Spatial relations • Activities • Communication • Verbal • Other • Tasks • How work is allocated

8. How to be an effective observer • Preparation • Stay in the background • Be factual and objective in your notes • (interpretation comes later) • Taking notes: • Hand written usually • Type in to computer later • EXPANDING NOTES (ASAP)

9. “(Participant) observation”: in natural setting • “Participant” observation occurs when you interact casually and/or form relationships with informants • How much you actually “participate” depends on the goals of the study.

10. Participant observation • Advantages: • Offers insights into complex behavior • Identify the “right questions” for further study • Verify/correct self-reports • Disadvantage: • Time consuming • Data collection is difficult • Problem of subjectivity

11. How to operationalize • Field notes • Text • Diagrams, maps • Can result in numerical data • Interviews (interviewer more clueful) • Focus groups (facilitator more clueful)

12. What to observe • Spatial relations • Activities • Communication • Verbal • Other • Tasks • How work is allocated • See Table 3 in reading

13. Ethics • Do not disrupt the activity your are observing versus • Do not mislead • No formal rules about disclosing your role as a researcher when engaging in casual conversation – article suggests a point where you want to ask specific question • Disclosure includes: right of refusal, confidentiality

14. Protecting confidentiality when data is unique • Separate identify info from field notes entered into the computer • People, organizations/companies, should be given fictitious names

15. How to be an effective observer • Preparation • Stay in the background • Be factual and objective in your notes • (interpretation comes later) • Taking notes: • Hand written usually • Type in to computer later • EXPANDING NOTES

16. Tips • Leave space • Take notes strategically • Use abbreviations • Cover a range of observations: Body language, etc.

17. Tips • Leave space • Take notes strategically • Use abbreviations • Cover a range of observations: Body language, etc.

18. Participant observation: in natural setting • “Participant” observation occurs when you interact and/or form relationships with informants • Demanding and time-consuming • How much you actually “participate” depends on the goals of the study. • Subjects may “forget” you are a researcher

19. How to operationalize direct/participant observation • Field notes • Text • Diagrams, maps • Can result in numerical data • Interviews (interviewer more clueful in P.O.) • Focus groups (facilitator more clueful in P.O.) “Water cooler” effect

20. Participant observation • Advantages: • Offers insights into complex behavior • Identify the “right questions” for further study • Verify/correct self-reports • Disadvantage: • Time consuming • Data collection is difficult • Problem of subjectivity

21. Ethics of direct observation/ participant observation • Do not disrupt the activity your are observing versus • Do not mislead • No formal rules about disclosing your role as a researcher when engaging in casual conversation – some authors suggest a point where you want to ask specific question • Disclosure includes: right of refusal, confidentiality

22. Protecting confidentiality when data is unique • Separate identify info from field notes entered into the computer • People, organizations/companies, should be given fictitious names

23. In-depth interview/focus group • Probes the interviewee(s) views of the phenomenon of interest • Interviewer/facilitator should be neutral • Data collected: transcript, audio/video recording, notes

24. In-depth interview/focus group Interviewer should: • Start with some open-ended questions • Follow up by asking “how” and “why” • Keep the discussion on track

25. Documents • Memos and meeting notes • Transcripts of conversations or speeches • Manuals and policy handbooks • Newspapers and magazines • Internet-based research • Email • Web sites • Blogs • Especially important in case studies

26. Used to analyze a written or spoken record for occurrence of specific behaviors or events Archival sources often used as sources for data Response categories must be clearly defined A method for quantifying behavior must be defined III. Artifacts: Content Analysis

27. The CEO of Global Enterprises, Inc. is very worried about the low morale in the company, as evidenced by the amount of flame email she receives. She considers sending every office on a “ropes” course, but to do this would cost the company $10M. She asks you to do a study to tell how well her scheme might actually work in reducing her flame mail. Example Study

28. Phenomenon tentatively defined Hypothesis is developed A single instance is considered to determine if hypothesis is confirmed If hypothesis fails, then phenomenon or hypothesis is redefined Additional cases are examined and, if the new hypothesis is repeatedly confirmed, some degree of certainty results Each negative case requires that the hypothesis be reformulated until there are no exceptions Analytic Induction Nonexperimental, Qualitative analogue to scientific method

29. Say you’re interested in employee’s impressions of WizziWord. You interview 3 people, transcribe your notes, and categorize all important statements into themes e.g. “It’s too slow.”, “It looks cool.”, etc. You interview 3 more people, categorize their comments. Repeat until no new (significant) categories/themes emerge. Typical Use of Analytic Induction

30. Compare/Integrate “all” studies that have investigated a given phenomena E.g., use of a particular medication for a particular disease Common in the literature (esp. medical) Very methodical Search for articles Eligibility criteria Statistical analyses Meta-Analyses

31. Case Study Research Steps: design, conduct (data collection), analysis, write-up Exploratory research and the role of prior theory Impacts case selection, data collection Scientific method – observations should have the potential to disconfirm Case study research questions Example: Telemedicine paper

32. Case study research Design steps Define unit of analysis Case selection Creation of a protocol (plan of work) Data Collection Analysis and reporting

33. Case Study Research I. Design steps: Select the unit(s) of analysis (temporal, organizational, technological) Case selection (“sampling”??) – one or several critical case theory based (confirming or disconfirming) extreme v. typical intense criterion (e.g., budget > $X) convenience

34. Case Study Research – Design steps (cont.) • Use of a protocol: (the “plan of work”, should be required) • Overview of study, including overview of data collection strategy. • Details of data collection (sources, procedures) • Interview guidelines and instruments • Outline of the expected project report • Issues to be addressed in (2): • access to the organization • resources sufficient to collect the data in the field • scheduling of data collection activities • providing for unanticipated events

35. Case Study Research – Data Collection The more different methods employed, the fuller the picture of the phenomena being studied. -- Documents (meeting minutes, project reports, newsletters, manuals) -- Archival documents (service records, system usage data) may provide quantitative information -- Interviews Typical: 95 interviews over 6 months -- Field observations (when a visit is conducted): usually meetings. (also can observe user training, etc.) -- Artifacts (problem reports – why not archival docs??)

36. Case Study Research – Data Collection Selecting interviewees: a. maximum variation (preferred method) b. Homogeneous c. Snowball or chain d. Purposeful v. opportunistic Benefits of semi-structured interviews Unstructured when questions not known in advance What is “triangulation”? (paper mentions construct validity) When to STOP collecting data

37. Case Study Research – Data Analysis Data analysis very different from analysis of experimentand survey data. Why? Stages of analysis: Preliminary analysis (early steps) Within-case analysis Cross-case analysis Qualitative analysis most difficult and least standardizedpart of empirical research.

38. Case Study Research – Data Analysis Goals of qualitative data analysis: Identify themes Develop categories Explore similarities and differences Describe patterns that explain why (Propose models that predict)

39. Case Study Research – Data Analysis Techniques for qualitative data analysis: Preliminary Stage 1. Coding 2. Database What is a code? a word or short phrase attached to each segment (e.g., paragraph, answer to interview question) of the collected data, indicating the “presence” of that concept. Codes can be arranged in (or derived from) a taxonomy. A good taxonomy will yield codes that reveal patterns in the data.

40. Case Study Research – Data Analysis Where do codes come from: Prior work or theory Study of initial data (defined “inductively”) Iterative nature of coding Use of independent raters to validate codes The code book or code manual: desirable attributes Detailed description of each code Inclusion and exclusion criteria Examples of collected data to illustrate each code Development of higher-level “pattern codes” identify themes or relationships that are relevant to the study’s research questions

41. Case Study Research – Data Analysis • Case study database: • uninterpreted data • complete data (answers criticism of “selective quoting”) • analogous to raw data collected in experiment or survey • Contents of case study database • Field notes (interviews, observations) • Documents (including transcripts) • Quantitative data (including questionnaire data if any) • Contemporaneous notes (reflective remarks)

42. Case Study Research – Data Analysis Techniques for qualitative data analysis: within-case stage Goal: identify larger themes, relationships and propositions Looking for larger themes and patterns: Pattern-matching compare expected elements with actual data perform cross-checking of interview transcripts and other data collected desire two or more sources for each proposition Explanation building: challenge  tactics  results approach Use of charts, table, graphs, timelines to aid understanding

43. Case Study Research – Data Analysis Techniques for qualitative data analysis: cross-case stage Depends on availability of several cases Two approaches: Analyze similarities and differences among cases (e.g., factors, behaviors, results) If goal is theory-building, develop a theory using one case and systematically compare its propositions to other cases

44. Case Study Research – Write-up • Weakest part of the article • Goals: • Includes the goals of all professional writing, e.g., clarity, shows relationship to earlier work, data support conclusions • For positivist case research, shows applicability (general relevance) to other examples with similar circumstances • Constructive – propositions translate into “lessons learned” that offer guidance on how to make use of the results (do’s and don’ts)

45. Qualitative Data Analysis by John V. Seidel • Description of how to go about analyzing transcripts of interviews, documents, and/or field notes. • Focus on “coding” • First identify “events” • Assign terms that represent concepts of interest • Organizing codes into a scheme • Building qualitative models using the coding scheme as the model vocabulary • Focus on iterative nature of QDA

46. Two perspectives on coding • Objectivist perspective • Condensed representation of facts • Can be subjected to hypothesis testing • Strong burden of consistency/completeness • Heuristic perspective • Signposts pointing to things you care about • Foundation for further analysis

48. Elements/types of qualitative models • Examples from Rogers’ theory of innovation diffusion • VCR’s • Cell phones • Metric system • Seat belts in cars • Dvorak keyboard

50. Three analogies to explain this • Jigsaw puzzle analogy • A little data and a lot of right brain • Multi-threaded DNA (patterns among the patterns)