Evaluation Research: Methods and Applications

1. IS 4800 Empirical Research Methods for Information Science Class Notes January 13, 2012 Instructor: Prof. Carole Hafner, 446 WVH hafner@ccs.neu.edu Tel: 617-373-5116 Course Web site: www.ccs.neu.edu/course/is4800sp12/

2. http://www.classictvads.com/smoke_1.shtml http://www.buzzfeed.com/jpmoore/x-ridiculous-vintage-cigarette-ads

3. Classification schemes for research methods • Epistemological – what kind of claims can be made • Descriptive/Interpretive • Correlational/Relational • Causal • Methodological: how we go about it (process) • Quantitative vs. qualitative; survey, experiment, etc. • Level of Generality (by purpose) • Scientific research aims to discover general principles or test a general theory • Evaluation research aims to support decision making or test a hypothesis in a specific context

4. Evaluation Research:Evaluating a product, solution, or method • Purpose: provide useful feedback about some application or technology (how well it achieves its goals, its value or impact, etc.) • Can use quantitative, qualitative or mixed methods • Supports a decision about what should be done • Best if using real-world (vs. laboratory) data • Pilot studies of systems or technologies enable evaluation research

5. Results of Evaluation Research • Choose which of several products to purchase • Decide whether or not to go forward with full deployment of a new system or process • Decide whether more or different training is needed • Choose which features of a product design should be kept and which ones dropped • Decide whether a policy or the governance structure that supports it needs to be changed • Decide whether a new or modified policy would be effective • etc.

6. Steps in evaluation research • Formulation of research questions/hypotheses • Selection of research strategy (quant. or qual.) • The usual three elements: • Population to observe • Characteristics to observe or measure • How to analyze the data • Utilization: what will be done, depending on the conclusions ? (an important question about medical tests ??)

7. Quantitative Research Claims • Quantitative research: the claims are supported by objective data and (usually) statistics • Descriptive studies – how are characteristics distributed in a population ? • Correlational studies – how are two characteristics related? • Inferential studies – do differences in one characteristic lead to differences in another characteristic?

8. Quantitative Research Claims: examples • People who use internet for research on medical issues ask more (average of 3.5) questions when they visited their doctor, while those who did not (average of 1.5 questions). • Employees at company ABC work faster with Word than with WizziWord.

9. Qualitative Research Claims • Qualitative research: claims are supported by observations and appeals to common sense (more “anecdotal”) • Descriptive: observations are organized into themes and patterns. (Ex: Rogers theory of innovation diffusion). • Exploratory: May identify hypotheses for later quantitative studies • Explanatory: offers the possibility of explaining why things happen in addition to what happens Each study by itself does not “prove” anything but if more and more studies show the same thing, it can be convincing.

10. Qualitative Research Claims: hypothetical example • Involvement in competitive sports among children: • When segregated by sex, there is a tendency for the boys to reject girls as friends and playmates in non-sport activities • When boys and girls participate together, there is an greater friendship and interaction between boys and girls in school and in other activities

11. Methodological classification • Quantitative methods (and type of claims) • Observational (descriptive and/or correlational) • Survey (descriptive and/or correlational) • Experiment (causal) • Quasi-experiment (weaker causal claims) • Qualitative methods • Ethnography (descriptive) • Case study (descriptive; explanatory) • Grounded theory (descriptive; exploratory) • Mixed methods

12. Quantitative Methods: Observational Studies • A population is identified; measures are defined • Data is collected by observing/recording, or from existing databases or other artifacts • Descriptive statistics and graphs reveal how the measures are distributed • Statistics may also reveal that there is a relationship between measures

13. Quantitative Methods: Observational Studies Example: Distribution by major of CCIS students Number/proportion of CCIS students in a combined major Relationship of gender to the above Conclusions: 42% of CCIS students choose combined major The data do (do not) reveal a relationship between gender and the choice of major

14. Quantitative Methods: Surveys • A population and a sampling technique are identified • A data collection instrument is created (usually a questionnaire) with questions representing self-report measures of interest. • Members of the sample are contacted and asked to answer the questions • Descriptive statistics and graphs reveal how the data are distributed • Graphs or correlation statistics may reveal that there is a relationship

15. Quantitative Methods: Surveys Example: 100 randomly selected recent NU graduates are asked to complete a survey about their post-graduation employment Conclusion: NU students with STEM degrees were more likely to have a full-time job within 6 months of graduating than students in other majors. (cannot claim causality)

16. Quantitative Methods: Experiments • A treatment variable (independent variable) is defined with at least two different values (levels), and an outcome measure (dependent variable) is also defined • A criterion level of statistical significance is selected • Participants are selected and assigned to treatment groups (randomly or using some balancing method) • The treatments are applied to the groups, and the outcomes are observed • Inferential statistics are used to determine if the data satisfy the criterion to conclude that different treatment levels lead to different outcomes.

17. Quantitative Methods: Quasi-experiments • A treatment variable (independent variable) is defined with at least two different values (levels), and an effect measure (dependent variable) is also defined • A criterion level of statistical significance is selected • Existing groups of participants are selected that are assumed or hoped to be comparable • The treatments are applied to groups, and the outcomes are observed • Inferential statistics are used to determine if the data satisfy the criterion to conclude that different treatment levels lead to different outcomes.

18. Advantages/disadvantages of quantitative methods +: Systematic rules and procedures already worked out, and can be followed +: Traditional, accepted as “proof” -: Closed-ended questions may lead to ignoring important factors and relationships -: Quantitative methods cannot handle phenomena that are difficult to turn into variables

19. Qualitative Methods: Ethnography • Purpose: Identify the social structures and behavior patterns of a culture (subgroups and roles; how status is achieved; how work gets done; distribution of resources; how, when and why people communicate) • Observations are made in the natural setting • The researcher often joins the society to experience directly how its members behave and interact

20. Qualitative Methods: Case Studies • Purpose: Explore in depth an activity or project • Purpose: Discover and tell a story with a beginning, middle and end • Multi-modal data collection (observation, interviews, artifacts . . .) • Identifies lessons learned that explain successful and unsuccessful outcomes • A frequently-used method for IT empirical research

21. Qualitative Methods: Case Studies Example: A failed software development project is studied to find out what lessons can be learned to avoid similar results in future projects

22. Advantages/disadvantages of qualitative methods +: more innovative and creative +: capable of addressing issues that do not lend themselves to being described by variables -: justification of conclusions from data is often less clear -: makes some physical scientists uncomfortable

23. Mixed Methods Pragmatic philosophy – find out whatever you can using whatever methods are possible Involves both qualitative and quantitative elements  (at least 2 stages of research) Advantages/Disadvantages +: combines structure and flexibility -: requires more time and resources

24. Examples of Mixed Method Designs Pattern 1: quantitative study followed by in-depth interviews to get insight on the reasons for the observed relationships. Typical in usability research. Pattern 2: exploratory qualitative study followed by a survey or experiment to confirm and generalize the results

25. Collecting data for quantitative studies Quantitative research skills Identify/define the variables (and their coding) Design the “instrument” – a measurement process or technique Types of instruments: a questionnaire (paper or Web-based) logs of “click” data form for observer to fill in (experiment or field study) Field testing/validating instruments is part of quantitative methods Real-world vs. laboratory data collection is a key distinction

26. Collecting data for qualitative studies Qualitative research skills Careful observation, being a good listener Gaining the trust of informants Linguistic analysis of documents and conversations Ability to recognize patterns in social behavior and in documents Types of instruments: Paper and pencil – making good field notes Audio and video recording Contemporaneous vs. retrospective information gathering is an important distinction

27. V. Purposes and Products • Reasons for doing empirical research • Disconfirm or support a theory • Solve a design or technical problem • Evaluate a product or solution • Inform practice (develop “best practices”) • Understand why people do what they do • Predict the future • Contribute to a better world • Advance personal goals

28. V. Purposes and Products • Products of Empirical Research • Evidence of a relationship • Disconfirm a theory (or some aspect of it) • Comparison of existing products or techniques • A new or improved product or technique • In-depth study of a particular outcome • Explanation of unanticipated outcomes • A new theory • A new improved research tool or technique • A critical analysis (pros and cons)