2006 Washington State Prevention Summit

1. 2006 Washington State Prevention Summit Analyzing and Preparing Data for Outcome-Based Evaluation Using the Assigned Measures and the PBPS Outcomes Report October 20, 2006 Sarah Stachowiak Organizational Research Services

2. Purpose and Goals • Increased knowledge of Assigned Measures (AMs) • Increased skills in collecting participant data • Increased skills in interpreting PBPS Outcomes Report

3. DASA Required Measures • Pre-Post Survey Questions for All Youth Participants 13-17 years old • PPG Items for Family, Community, School and Individual Domains • Questions on: Perceived Risk, Perceived Harm, Perceived “Wrongfulness” and 30-Day Use of Substances • 15 Questions on PPG03 – Individual Domain Scale

4. DASA Assigned Measures – Development Process • Search of literature on Impacts and Effects of Different Best Practices • Search for common shorter term, more direct outcomes for youth and parents participating in different programs and practices • Definitions of Outcomes -> Measurable Indicators • Search for Valid and Reliable Measurement Scales

5. DASA Assigned Measures • Pre-Post Survey Questions for All Youth Participants 13-17 years old or All Parents/Guardians • Set of Youth and Parent Outcomes that are aligned with different Best and Promising Practices (9 Youth Outcomes / 8 Parent Outcomes) • Scales with 5-8 questions for each of the Assigned Measures – drawn from existing tools or scales

6. Measurement Scales • Search Through Validated Instruments and Curriculum Surveys • Identified Survey Items Consistent with Chosen Indicators Linked to Youth and Parent Outcomes • 5-10 Additional Survey Questions per Outcome • Data Collection Across Programs Addressing Outcome and Objectives

7. Parent Outcomes • Improved Family Cohesion • Improved Attitudes about Family Management Skills • Increased Use of Family Management Skills • Increased Family Involvement • Improved Family Communication • Reduced Family Conflict

8. Youth Outcomes • Improved Bonding • Less Favorable Attitudes • Increased Refusal/Resistance Skills • Improved Social Competence Skills • Improved Personal Competence • Reduced Anti-Social Behaviors • Improved Academic Performance

9. Benefits of Assigned Measures • More useful outcome data for County/Tribe and Provider purposes • Ability to look at common changes across different Best Practices and other Programs • More “realistic” questions for respondents • Now have parent outcome data!!

10. Collecting Participant Data • Participant ID Issues • Administering Surveys • Managing Data Collection

11. Assigning ID Numbers • Track participants over time • Administer a multiple tools (e.g., pre and post) • Confidentiality versus anonymity • Unique identifiers • Simple ID • Self-Generated ID • Local ID Field in PBPS

12. Self-Generated ID Numbers • What is the last letter of your first name? • What is the second letter of your last name? • What is the month of your birthday? • What is the first letter of your middle name?

13. Administering Surveys • Share the purpose and intent • Assure confidentiality • Make sure everyone understands the ID code directions • Consider type of administration (e.g., facilitator reads questions)

14. Managing Data Collection • Maintain a survey tracking system • Take steps to maximize response rate • Use “data windows” • Collect data when you have access to participants • Consider incentives

15. PBPS Outcome Report • Levels of Aggregation • Types of Data Presented • Service Characteristics • Pre-Post Changes

16. Levels of Aggregation

17. Descriptive Data • Frequencies: summaries of the number or percent of observations in each response category • Averages: mean of responses • Cross-tabulations: summaries of frequency distributions across different subgroups or levels of a second variable (not yet available)

18. T-Tests • Test for statistically significant difference between mean values • Paired Samples – comparison of mean values on one variable over time for the same participants (e.g., Pre vs. Post) • Mean differences “not due to chance” • Standard convention p <.05 (probability that difference is due to chance is less than 5 percent)

19. Interpreting Quantitative Data Look at your data: • What patterns do you see in the rows and columns? • What findings are most interesting? • What client characteristics might explain these patterns? • What program strategies might explain these patterns?

20. Service Characteristics/Demographics • Survey Completion Rate • Average Attendance Rate • Frequencies for: • Gender • Race • Ethnicity • Age (not for parent programs) • Note: Data are dynamic; only relevant categories are shown • Note: Demographics for all participants, not those who had pre-post data

21. Question Detail • Scoring scale • # Pre Post • Pre and Post Results • Average scores • Statistical Significance • Better, Worse, No Change • % Change • State Comparison • Sub-Scales/Average of Questions • #/% Individuals whose scores were…

22. Interpretation Considerations • Sample size • Completion rate • Representativeness • Cross tabulations (available 2007)

23. Group Exercise • Interpreting Outcome Report Data

24. Reporting Findings Considerations: • What do the data say about the outcomes? • Who is your audience? What is your purpose? • How can you best communicate what the data say? • What are the implications of the findings for program development? For marketing?

25. Reporting Findings • Provide Context: • Outputs (e.g., dosage (frequency, quantity of intervention, number of participants) • Description of intervention • Background information that will help you interpret the data • Process information (e.g., fidelity)

26. Resources • Updated Evaluation Guidebook • Regional Prevention Managers

27. Final Thoughts • Goals of AMs and Outcome Report: • Learning! • Better decision-making • Stronger prevention planning and programming • Work in progress

28. Contact Information Sarah Stachowiak Organizational Research Services sarahs@organizationalresearch.com 206-728-0474 x10