Module Objectives

1. QUALITATIVEMETHODS IN PROGRAMEVALUATIONBrian J. English Ph.D.International Public Policy and Management ProgramUniversity of Southern California

2. “Discovery consists in seeing what everyone else has seen and thinking what no one else has thought.”Albert Szent-Gyorgi,1937 Nobel Prize Physiology and Medicine

3. Module Objectives • What is qualitative research? • How do quantitative research and qualitative research differ? • What qualitative methods are appropriate for program evaluation? • Why are qualitative methods appropriate for gathering data to evaluate public policy and intervention programs?

4. Module Objectives • How can qualitative and quantitative methods complement each other? • How can qualitative data aid in the planning and implementation of intervention programs? NOTE: This module will not cover data coding or the analysis of qualitative data.

5. Defining Qualitative Research “A particular tradition in social science that fundamentally depends on watching people in their own territory and interacting with them in their own language, on their own terms.” (Kirk and Miller, 1986, p.9)

6. Defining Qualitative Research "Qualitative research is a loosely defined category of research designs or models, all of which elicit verbal, visual, tactile, olfactory, and gustatory data in the form of descriptive narratives like field notes, recordings, or other transcriptions from audio- and videotapes and other written records and pictures or films.“ Judith Preissle

7. Brian’s Definitions • Interpretive • Naturalistic • Can be Ethnographic (a specific type) • Descriptive • Uses all 5 (or 6) of the senses • Writers’ research

11. How do Quantitative and Qualitative Research differ? • Qualitative researchers try to understand reality as it is construed by the persons being studied. • Qualitative researchers try to understand the meaning of particular activities or beliefs in the context of the culture being considered (Valadez and Bamberger, 1994)

12. Qualitative Inductive Subjective/insider centered Process oriented Lack of control of variables Quantitative Deductive Objective/outsider centered Outcome oriented Attempt to control variables

13. Qualitative Goal: Understand actors view Discovery Oriented Explanatory Quantitative Goal: Find facts and causes Verification Oriented Confirmatory

14. Differences in Data

15. What Qualitative Research Methods are appropriate for gathering data for Public Policy Program Evaluation? • Document analysis of archival records • Observation & Participant Observation • Interviews/Focus groups • Chismoso & Laquachero • Triangulation

16. Document Analysis • Newspapers • Birth/death records • Environmental Impact Reports • Financial Reports • Previous Evaluations • Journals • Film & Photos (Grand Canyon example)

17. “You can observe a lot just by watching." --YogiBerra

18. Observation • Gaining Access • Participant Observation • Etic vs. Emic • Shadowing

19. Interviews/Focus Groups • Key Informants • Confidential Informants • Informal Interviews • Structured Interviews

20. Chismoso & Laquachero • Talking and Walking • Need an Outgoing, Communicative Approach • Gaining trust • Note-taking & recordings can frighten or turn-off potential informants

21. Triangulation Document Analysis Interviews Observation

22. Why are qualitative methods appropriate for gathering data to evaluate public policy and intervention programs?&How can qualitative and quantitative methods complement each other?

23. The intricate relationship between human behavior and public policies can restrict or promote specific behaviors. Examples: • Restrictions on fishing • Prescription Drug policies • Zoning Legislation

24. The connections between public policy, infrastructure planning, environmental degradation, health, education and poverty • East Coast Land Developers • Waste Management and Dengue Fever • Coastal Resource Management

25. Numbers do not always reflect reality • Ivy tower syndrome • Glass ceiling syndrome • US prisons • Money drives policy; Money drives research

26. The data collected in qualitative studies can be useful in developing quantitative studies. • Determining Indicators • Identifying factors/variables • Adding Perspective

27. The learning process and behavioral changes that take place during a program intervention may best be explored in a natural setting over a long period of time. • Crosscheck information gathered in quantitative surveys • Aid in planning educational activities • Aid in Program revision

28. How can data aid in the planning and implementation of intervention programs? • Qualitative methods can reveal how the local population feels about a new project and how that project may affect their lives. • Program designers have a moral obligation to try to understand how locals feel about interventions.

29. Strengths Depth & Detail Openness Insider Perspective Tries to avoid pre-judging Can be less expensive Weaknesses Smaller Sample Less easily generalized Difficult to make systematic comparisons Dependant on researchers skills The effect of the researcher’s participation

30. Considerations for Further Thought • Unobtrusiveness • Fitting in • Ethical Issues • Legal Issues • Coding • Analysis • Drawing Inferences • Validity

31. Comments &Questions