CRJS 4466 PROGRAM & POLICY EVALUATION LECTURE #5

1. CRJS 4466 PROGRAM & POLICY EVALUATION LECTURE #5 • Evaluation projects • Questions?

2. Qualitative Evaluation Methods

3. Qualitative Methods – strengths • Carol Weiss (1998) – the coming of age of qualitative methods • note the Campbell versus Cronbach controversy here • qualitative methods are an important technique for under- • standing the ‘context’ of success or failure of programs • qualitative methods most commonly used in process evaluation • (formative) or the ‘how’ and ‘why’ of a program and its • workings • ability to represent subtleties and complexities of program • functioning and dynamics

4. Qualitative Methods - strengths • many evaluators like the interpersonal nature of qualitative • methodologies • qualitative approaches are often ‘action oriented’ • a choice of ‘depth’ over ‘breadth’

5. Qualitative Methods - limitations • reliability and validity of measures • interviewer bias • anecdotal • labour-intensive • costly • less useful for reporting on program outcomes

6. Qualitative Methods – Single System Evaluations • evaluating an intervention with a single client system, usually • quantitatively during the course of treatment • limited or questionable usefulness of qualitative methodology

7. 4. Qualitative Methods – Focused Qualitative Evaluation (FQE) • reflects the immediacy of much evaluation – not time for • extended research, ethnomethodological development • use of FG and rapid ethnographic assessment

8. Qualitative Methods – Mixed Methods • ‘triangulation’, combining both quantitative and qualitative • methods • ‘temporal sequencing’ and qualitative subsamples • benefit of both breadth and depth

9. Qualitative Methods – Action (Participatory) Research • overtly ‘action-oriented’ approach primarily among • those conducting qualitative evaluations • participatory, collaborative, empowering – now an expectation • in some evaluation projects • ‘praxis’ orientation

10. Qualitative Methods – Research Design • units of analysis and comparative designs • gaining access, key informants, and sponsorship • reciprocity, payback and feedback – and the methodological, • ethical concerns here (‘false hope’) • sampling considerations • deviant case sampling/typical case sampling • maximum variation sampling • snowball sampling • purposive sampling • convenience sampling • random sampling

11. Qualitative Methods – Research Design • data collection strategies • on-site observation • participant observation • reflexivity • use of interview guide • use of focus groups • cross-validation among interviewers • use of documents • when to stop data collection – the issue of saturation

12. Qualitative Methods – Data Management and Analysis • use of qualitative data analysis programs: Ethnograph, AskSam • QSR (Nudist) • coding data • emic (indigenous) and etic (researcher-created) coding • quality control mechanisms • reporting – issues related to qualitative methodologies

13. Table 5-1: Key Differences Between Qualitative and Quantitative Evaluation Approaches

14. Table 5.2: Summary of Key Questions in Conducting Qualitative Evaluation Assessments and Evaluation Studies

15. Table 5-3: Sampling Strategies for Qualitative Evaluations

16. Table 5-4: Open Ended Questions for the Economic Development Agreement Stakeholders Project

17. Table 5-4: Open Ended Questions for the Economic Development Agreement Stakeholders Project

18. Table 5-5: Some Basics of Face-to-Face Interviewing

19. Table 5-6: Helpful Hints as You Analyze Qualitative Data

20. Table 5-6: Helpful Hints as You Analyze Qualitative Data

21. Table 5-7: Comparing Qualitative Validity with Experimental and Quasi-Experimental Validity

22. Table 5-8: Ways of Testing and Confirming Qualitative Findings

23. Focus Group Question: “Based on your own experiences of the Criminal Justice Program here at Nipissing University, what do you like about the program? What don’t you like about the program?