CRJS 4466 PROGRAM & POLICY EVALUATION Qualitative Methods in Evaluation

1. CRJS 4466 PROGRAM & POLICY EVALUATION Qualitative Methods in Evaluation 1. Test on Wednesday 2. Evaluation Project – Questions?

2. 4. Qualitative Methods – strengths • Carol Weiss (1998) – the coming of age of qualitative methods • note the Campbell versus Cronbach controversy here • qualitative methods are an important technique for under- • standing the ‘context’ of success or failure of programs • qualitative methods most commonly used in process evaluation • or the ‘how’ and ‘why’ of a program and its workings • also common in formative evaluations • ability to represent subtleties and complexities of program • functioning and dynamics

3. 4. Qualitative Methods - strengths • many evaluators like the interpersonal nature of qualitative • methodologies • qualitative approaches are often ‘action oriented’ • a choice of ‘depth’ over ‘breadth’

4. 4. Qualitative Methods - limitations • reliability and validity of measures • interviewer bias • anecdotal • labour-intensive • costly • less useful for reporting on program outcomes

5. 4. Qualitative Methods – Single System (SS) Evaluations • evaluating an intervention with a single client system, usually • quantitatively during the course of treatment • limited or questionable usefulness of qualitative methodology • qualitative methods and SS designs best matched at • community or system level, where emphasis is on • understanding the context (policy space) rather than on a • single program/intervention

6. 4. Qualitative Methods – Focused Qualitative Evaluation (FQE) • reflects the immediacy of much evaluation – not time for • extended research, ethnomethodological development – instead, • ‘the best answer you can give me, under the circumstances’ • use of FG and rapid ethnographic assessment

7. 4. Qualitative Methods – Mixed Methods • ‘triangulation’, combining both quantitative and qualitative • methods • ‘temporal sequencing’ and qualitative subsamples • benefit of both breadth and depth

8. 4. Qualitative Methods – Action (Participatory) Research • overtly ‘action-oriented’ approach primarily among • those conducting qualitative evaluations • participatory, collaborative, empowering – now an expectation • in some evaluation projects • ‘praxis’ orientation

9. 5. Qualitative Methods – Research Design • units of analysis and comparative designs • gaining access, key informants, and sponsorship • reciprocity, payback and feedback – and the methodological, • ethical concerns here (‘false hope’) • sampling considerations • deviant case sampling (outliers) /typical case sampling • maximum variation sampling • snowball sampling • purposive sampling • convenience sampling • random sampling

10. 5. Qualitative Methods – Research Design • data collection strategies • on-site observation • participant observation • reflexivity • use of interview guide • use of focus groups • cross-validation among interviewers • use of documents • when to stop data collection – the issue of saturation

11. 5. Qualitative Methods – Data Management and Analysis • use of qualitative data analysis programs: Ethnograph, AskSam • QSR (Nudist) • coding data • emic (indigenous) and etic (researcher-created) coding • quality control mechanisms • reporting – issues related to qualitative methodologies