100 likes | 195 Views
Objectives for Session Eleven. Turn in Data Assignment Discuss Hypothetical Formative Evaluation Design Analyzing Qualitative Data Report Writing, Organization, and Utilization Strengths and Weaknesses of Different Data Collection Strategies.
E N D
Objectives for Session Eleven • Turn in Data Assignment • Discuss Hypothetical Formative Evaluation Design • Analyzing Qualitative Data • Report Writing, Organization, and Utilization • Strengths and Weaknesses of Different Data Collection Strategies
STOP-DWI in NY State • STOP-DWI is implemented in NY to reduce drunken-driving and alcohol-related accidents • Program theory • Implementation/Service Delivery • Checkpoints, tougher penalties, public service announcements, and education programs • How would we know if it was being implemented well?
Evaluation Design Proposal for STOP-DWI in NY State • Program Description • Who, where, what, how • Steps in delivery, logic model, resources/inputs • Evaluation Questions and Justification • Data Collection Strategy • Focus on 4 stages: checkpoints, adjudication in courts, advertisements, and education programs • Select methods that are appropriate for given stage • Mix of methods – quantitative and qualitative • Validity and reliability • Sampling
Analyzing Data • Approach to Quantitative Analysis • Missing data and Outliers • Description of Respondents • Dimensions of Interest/Relationships among Key Variables – move to more sophisticated analysis • Approach to Qualitative Analysis • Confidence in your data? • What do you know about the program? • What questions or relationships are you most interested in? What does your data tell you about these questions or relationships?
Strengths and Weaknesses of Different Qualitative Techniques • Differences across observation techniques • Differences between focus groups and interviews • Document review = highly variable • Note that a variety of techniques are useful in a formative evaluation setting
Integrating Qualitative and Quantitative Data • New programs • Program effects or measures are hard to quantify • Qualitative data to provide insight into or corroborate quantitative findings • Demands of client and context
Report Writing • Description of the Program • Presentation of Analysis • Implemented as planned or intended? • What affected implementation? • Interpretation of the Findings • What do your data suggest or mean for the stakeholders? • Justify and support with data • Conclusions and Recommendations • Prioritize, feasibility
Factors Undermining Utility of Evaluation Reports • Poor quality • Lack generalizability • Lack program description • Mismatch between research questions and questions data can answer • Not completed on time • Characteristics of commitment of organization
Readings for Next Time • Leon, Dziegielewski, and Tubiak. “A Program Evaluation Of A Juvenile Halfway House: Considerations For Strengthening Program Components.” • D’Emidio-Caston and Brown. “The Other Side Of The Story: Student Narratives on the California Drug, Alcohol, and Tobacco Education Programs.”