READY - A Youth Development Outcomes Measure

1. Rochester Evaluation of Asset Development for Youth READY - A Youth Development Outcomes Measure

2. Project Team • United Way of Greater Rochester • Kathy Lewis • Elizabeth Ramsay • Rochester-Monroe County Youth Bureau • Chris Dandino • Univ. of Rochester Medical Center, Div. of Adolescent Medicine • Jonathan D. Klein, MD, MPH • Melissa Matos Auerbach, MA • Shannon M. Smith • Sheryl Ryan, MD • Cheryl Kodjo, MD, MPH • Premini Sabaratnam, MPH • Representatives from youth serving agencies in Rochester

3. Participating Agencies included: • Baden Street Settlement • Big Brothers, Big Sisters of Greater Rochester • Boy Scouts of America, Otetiana Council • Boys & Girls Club of Rocehster • Center for Youth Services • Charles Settlement • Community Place of Greater Rochester • Girl Scouts of Genesee Valley • Hochstein Music School • Metro Council for Teen Potential • Southwest Area Neighborhood Association (SWAN) • Urban League of Rochester • YMCA of Greater Rochester

4. Youth Development • Philosophy or approach - a set of principles emphasizing active support of the growing capacity of young people by individuals, organizations, and institutions • Characterized by a positive, strength building orientation • Occurs at home, in school, among peer groups, and in community-based programs • Has gained importance nationally, in states, and in local communities

5. Community Outcomes • Local funders are increasingly concerned with demonstrating effective progress toward outcomes • Existing measures of youth development are lengthy and complex

6. Youth Development Outcomes Measurement Project • GOAL: • To develop an evaluation tool for YD programs that met the following criteria: • Easy to Use • Easy to Administer • Applicable to a Variety of Youth Development Programs • Useful for Assessment of Impact of Program on Youth Development of Participants

7. Rochester, New York • City in Monroe County, Western New York • Population 219,773 • 52% non-white • 37% ages 19 or less • Person under age 18 in 34% of households • 32% of families with children under 18 below poverty level Source: US Census, 2000

8. Youth-Serving Agencies Serve the Youth of Rochester through: • case management • counseling • homework assistance • sports programs • life skills building • leadership programs • music lessons • provision of safe, open recreational spaces

9. Three Phase Project • Phase I Instrument development via a consensus process • Phase II Piloting to test validity and reliability of instrument • Phase III Field tests and dissemination

10. Phase I: Instrument Development -Dec. 2000 - May 2001 • Meetings with representatives from youth-serving agencies and funders • Identification of core and optional outcome measures and questionnaire items • Establishment of face validity of core measures and measurement strategies

11. Identification of Core Outcomes • Three meetings resulted in list of 54 indicators and 10 outcomes • peer and adult relationships • constructive use of leisure time • basic social skills • community service • health maintenance • decision making process • responsibility • understanding boundaries/rules • positive identity • independent/daily skills

12. Narrowing List of Outcomes Agency representatives were asked: • What impact does your program have? • What would you like to learn to improve the quality of your program?

13. Programs Wanted to Know • Effectiveness of staff • Effectiveness of services they provide • Impact on youth and their families • Impact of youth involvement in more than one program • Youth Development philosophy of staff • Gaps and what programs can do about them

14. Programs used nominal iterative process to identify consensus priority areas for youth that they could impact First Round: 18 constructs Second Round: 7 constructs Top 2 retained Third Round: 4 constructs Top 2 retained Consensus Process

15. Outcomes for Operationalization • Basic Social Skills • Caring Adult Relationships • Decision Making Process • Constructive Use of Leisure Time

16. Candidate Questions Questions adapted from instruments by • Add Health • Boys and Girls Club of America • Girl Scouts of America • Metro Council for Teen Potential • Worcester Youth Development Initiative • YMCA

17. Phase II: Piloting the Draft Instrument -May 2001-March 2002 Piloting in two phases: • Cognitive interviews to test validity of items • Field test of internal consistency of items and feasibility

18. A. Cognitive Interviews • 48 urban and suburban adolescents aged 10 to 17 • Mean completion time: 11 minutes • 70% had no suggestions • 67% reported survey was “easy” to complete • 81% understood everything in the survey • 98% did not mind answering the survey • Items re-worded to increase readability (now at 4th grade level) and to simplify concepts

19. B. Field Test • 389 urban and suburban adolescents • Ages 10 to 19 • Large drop-in programs and smaller, structured programs • Findings: • Feasible for program staff to administer • Large groups required more staff time • Easy for older adolescents • Some issues remain for younger adolescents

20. Field Test Results • Youth more attached to programs did better on measures • Four constructs have several good factors for program use in evaluation • Instrument consists of six factors, corresponding to three outcomes • Internal reliability scores (as from .5782 to .8557)

21. Factor Analysis: Core Outcomes Self Control Empathy Basic Social Skills Communication Staff Relationships Caring Adult Relationships Program Effect Decision Making Decision Making Constructive Use of Leisure Time

22. Factor Analysis - all participants

23. Factor Analysis - 13-19 year olds

24. Factor Analysis - Reliability

25. Phase II: Con’t • Slight revision of individual items and rearrangement of questions leading to final instrument: • Pencil and paper survey • 40 questions addressing four core outcomes, program participation, connectedness to program, and sociodemographic information • Requires between 10 and 15 minutes to complete • Development of training modules, scoring templates, and score report generating software

26. Phase III: Dissemination in Rochester - May 2002-May 2003 • Summer and Fall 2002 - 11 youth serving agencies in the Rochester area were trained to use the instrument and the report generating software • TA provided to agencies to develop appropriate sampling plans • During the program year of 2002-2003, over 1,000 youth participating in YD programs in the Rochester area completed surveys

27. Current Steps:May 2003 - present • Software and score reports revised based on qualitative feedback from Year 1 implementation • Continued training and TA to current users • Dissemination to various other youth development programs through ACT for Youth Center of Excellence • Validation studies

28. Additional sites • Alaska - Residential School System served as a beta test site in school year ‘02-’03 • Erie County, NY - over 75 programs including youth bureaus and youth boards, and UW and Department of Youth Services funded programs • Hawaii - Children’s Alliance of Hawaii • Oswego County, NY - Oswego City-County Youth Bureau funded programs • Salamanca, NY - 21st CCLC program • Syracuse, NY - Catholic Charities of Onondaga County sites • Queens, NY - Queens Child Guidance Center Beacon & OST sites

29. Use of Data by Programs • Internal quality improvement. Examples include: • Reviewing and discussing score reports with staff and with Boards • Comparing program scores within one agency to identify opportunities for improvement • Reviewing curricula and current programming strategies • Discussing program strategies with other similar programs • Identifying training and technical assistance needs • Reporting to funders • Proposal writing • Sharing data back allowing the creation of an aggregate community level score report using de-identified data

30. Training and Technical Assistance • Training, TA, and use of the READY Toolkit are available to interested users • Fees are based on the number of users and the level of training and TA required • Options include: • Training and TA provided directly to end users • One time training provided for end users, and continuous training and TA provided to a lead agency which then agrees to provide first line TA to end users

31. READY Toolkit • READY Toolkit includes a CD which contains: • A Personalizable Instrument Template • READY Analysis Program • Toolkit Instructions Manual • User’s Agreement • Instrument template may be personalized to contain program names and staff titles • READY Analysis program allows community programs to enter their own survey data, and generate a score report • Score report contains summary measures for core YD outcomes and frequencies for all survey items

32. SAMPLE Score Report Pg 1 of 17

33. SAMPLE Score Report Pg 2 of 17

34. Publications • Klein JD, Sabaratnam P, Matos Auerbach M, Smith SM, Kodjo C, Lewis K, Ryan S, Dandino C. Development and factor structure of a brief instrument to assess the impact of community programs on positive youth development: The Rochester Evaluation of Asset Development for Youth (READY) tool. Journal of Adolescent Health 2006; 39: 252-260. • Sabaratnam P, Klein JD. Measuring youth development outcomes for community program evaluation and quality improvement: Findings from dissemination of the Rochester Evaluation of Asset Development for Youth (READY) Tool. Journal of Public Health Management and Practice 2006; 6(suppl): S88-S94.

35. For more information about the READY tool, please contact: Premini Sabaratnam, MPH Sr. Health Project Coordinator Div. of Adolescent Medicine, University of Rochester (585) 273-4616 Premini_Sabaratnam@urmc.rochester.edu or Jonathan D. Klein, MD, MPH Associate Professor of Pediatrics and of Community & Preventive Medicine University of Rochester (585) 275-7760 Jonathan_Klein@urmc.rochester.edu

36. Or visit... The University of Rochester, Division of Adolescent Medicine, Leadership Education in Adolescent Health website at www.urmc.rochester.edu/gchas/div/adol/leah/resources.htm or The ACT for Youth, Center of Excellence website at www.actforyouth.net