1 / 36

READY - A Youth Development Outcomes Measure

Rochester Evaluation of Asset Development for Youth. READY - A Youth Development Outcomes Measure. Project Team. United Way of Greater Rochester Kathy Lewis Elizabeth Ramsay Rochester-Monroe County Youth Bureau Chris Dandino Univ. of Rochester Medical Center, Div. of Adolescent Medicine

thais
Download Presentation

READY - A Youth Development Outcomes Measure

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Rochester Evaluation of Asset Development for Youth READY - A Youth Development Outcomes Measure

  2. Project Team • United Way of Greater Rochester • Kathy Lewis • Elizabeth Ramsay • Rochester-Monroe County Youth Bureau • Chris Dandino • Univ. of Rochester Medical Center, Div. of Adolescent Medicine • Jonathan D. Klein, MD, MPH • Melissa Matos Auerbach, MA • Shannon M. Smith • Sheryl Ryan, MD • Cheryl Kodjo, MD, MPH • Premini Sabaratnam, MPH • Representatives from youth serving agencies in Rochester

  3. Participating Agencies included: • Baden Street Settlement • Big Brothers, Big Sisters of Greater Rochester • Boy Scouts of America, Otetiana Council • Boys & Girls Club of Rocehster • Center for Youth Services • Charles Settlement • Community Place of Greater Rochester • Girl Scouts of Genesee Valley • Hochstein Music School • Metro Council for Teen Potential • Southwest Area Neighborhood Association (SWAN) • Urban League of Rochester • YMCA of Greater Rochester

  4. Youth Development • Philosophy or approach - a set of principles emphasizing active support of the growing capacity of young people by individuals, organizations, and institutions • Characterized by a positive, strength building orientation • Occurs at home, in school, among peer groups, and in community-based programs • Has gained importance nationally, in states, and in local communities

  5. Community Outcomes • Local funders are increasingly concerned with demonstrating effective progress toward outcomes • Existing measures of youth development are lengthy and complex

  6. Youth Development Outcomes Measurement Project • GOAL: • To develop an evaluation tool for YD programs that met the following criteria: • Easy to Use • Easy to Administer • Applicable to a Variety of Youth Development Programs • Useful for Assessment of Impact of Program on Youth Development of Participants

  7. Rochester, New York • City in Monroe County, Western New York • Population 219,773 • 52% non-white • 37% ages 19 or less • Person under age 18 in 34% of households • 32% of families with children under 18 below poverty level Source: US Census, 2000

  8. Youth-Serving Agencies Serve the Youth of Rochester through: • case management • counseling • homework assistance • sports programs • life skills building • leadership programs • music lessons • provision of safe, open recreational spaces

  9. Three Phase Project • Phase I Instrument development via a consensus process • Phase II Piloting to test validity and reliability of instrument • Phase III Field tests and dissemination

  10. Phase I: Instrument Development -Dec. 2000 - May 2001 • Meetings with representatives from youth-serving agencies and funders • Identification of core and optional outcome measures and questionnaire items • Establishment of face validity of core measures and measurement strategies

  11. Identification of Core Outcomes • Three meetings resulted in list of 54 indicators and 10 outcomes • peer and adult relationships • constructive use of leisure time • basic social skills • community service • health maintenance • decision making process • responsibility • understanding boundaries/rules • positive identity • independent/daily skills

  12. Narrowing List of Outcomes Agency representatives were asked: • What impact does your program have? • What would you like to learn to improve the quality of your program?

  13. Programs Wanted to Know • Effectiveness of staff • Effectiveness of services they provide • Impact on youth and their families • Impact of youth involvement in more than one program • Youth Development philosophy of staff • Gaps and what programs can do about them

  14. Programs used nominal iterative process to identify consensus priority areas for youth that they could impact First Round: 18 constructs Second Round: 7 constructs Top 2 retained Third Round: 4 constructs Top 2 retained Consensus Process

  15. Outcomes for Operationalization • Basic Social Skills • Caring Adult Relationships • Decision Making Process • Constructive Use of Leisure Time

  16. Candidate Questions Questions adapted from instruments by • Add Health • Boys and Girls Club of America • Girl Scouts of America • Metro Council for Teen Potential • Worcester Youth Development Initiative • YMCA

  17. Phase II: Piloting the Draft Instrument -May 2001-March 2002 Piloting in two phases: • Cognitive interviews to test validity of items • Field test of internal consistency of items and feasibility

  18. A. Cognitive Interviews • 48 urban and suburban adolescents aged 10 to 17 • Mean completion time: 11 minutes • 70% had no suggestions • 67% reported survey was “easy” to complete • 81% understood everything in the survey • 98% did not mind answering the survey • Items re-worded to increase readability (now at 4th grade level) and to simplify concepts

  19. B. Field Test • 389 urban and suburban adolescents • Ages 10 to 19 • Large drop-in programs and smaller, structured programs • Findings: • Feasible for program staff to administer • Large groups required more staff time • Easy for older adolescents • Some issues remain for younger adolescents

  20. Field Test Results • Youth more attached to programs did better on measures • Four constructs have several good factors for program use in evaluation • Instrument consists of six factors, corresponding to three outcomes • Internal reliability scores (as from .5782 to .8557)

  21. Factor Analysis: Core Outcomes Self Control Empathy Basic Social Skills Communication Staff Relationships Caring Adult Relationships Program Effect Decision Making Decision Making Constructive Use of Leisure Time

  22. Factor Analysis - all participants

  23. Factor Analysis - 13-19 year olds

  24. Factor Analysis - Reliability

  25. Phase II: Con’t • Slight revision of individual items and rearrangement of questions leading to final instrument: • Pencil and paper survey • 40 questions addressing four core outcomes, program participation, connectedness to program, and sociodemographic information • Requires between 10 and 15 minutes to complete • Development of training modules, scoring templates, and score report generating software

  26. Phase III: Dissemination in Rochester - May 2002-May 2003 • Summer and Fall 2002 - 11 youth serving agencies in the Rochester area were trained to use the instrument and the report generating software • TA provided to agencies to develop appropriate sampling plans • During the program year of 2002-2003, over 1,000 youth participating in YD programs in the Rochester area completed surveys

  27. Current Steps:May 2003 - present • Software and score reports revised based on qualitative feedback from Year 1 implementation • Continued training and TA to current users • Dissemination to various other youth development programs through ACT for Youth Center of Excellence • Validation studies

  28. Additional sites • Alaska - Residential School System served as a beta test site in school year ‘02-’03 • Erie County, NY - over 75 programs including youth bureaus and youth boards, and UW and Department of Youth Services funded programs • Hawaii - Children’s Alliance of Hawaii • Oswego County, NY - Oswego City-County Youth Bureau funded programs • Salamanca, NY - 21st CCLC program • Syracuse, NY - Catholic Charities of Onondaga County sites • Queens, NY - Queens Child Guidance Center Beacon & OST sites

  29. Use of Data by Programs • Internal quality improvement. Examples include: • Reviewing and discussing score reports with staff and with Boards • Comparing program scores within one agency to identify opportunities for improvement • Reviewing curricula and current programming strategies • Discussing program strategies with other similar programs • Identifying training and technical assistance needs • Reporting to funders • Proposal writing • Sharing data back allowing the creation of an aggregate community level score report using de-identified data

  30. Training and Technical Assistance • Training, TA, and use of the READY Toolkit are available to interested users • Fees are based on the number of users and the level of training and TA required • Options include: • Training and TA provided directly to end users • One time training provided for end users, and continuous training and TA provided to a lead agency which then agrees to provide first line TA to end users

  31. READY Toolkit • READY Toolkit includes a CD which contains: • A Personalizable Instrument Template • READY Analysis Program • Toolkit Instructions Manual • User’s Agreement • Instrument template may be personalized to contain program names and staff titles • READY Analysis program allows community programs to enter their own survey data, and generate a score report • Score report contains summary measures for core YD outcomes and frequencies for all survey items

  32. SAMPLE Score Report Pg 1 of 17

  33. SAMPLE Score Report Pg 2 of 17

  34. Publications • Klein JD, Sabaratnam P, Matos Auerbach M, Smith SM, Kodjo C, Lewis K, Ryan S, Dandino C. Development and factor structure of a brief instrument to assess the impact of community programs on positive youth development: The Rochester Evaluation of Asset Development for Youth (READY) tool. Journal of Adolescent Health 2006; 39: 252-260. • Sabaratnam P, Klein JD. Measuring youth development outcomes for community program evaluation and quality improvement: Findings from dissemination of the Rochester Evaluation of Asset Development for Youth (READY) Tool. Journal of Public Health Management and Practice 2006; 6(suppl): S88-S94.

  35. For more information about the READY tool, please contact: Premini Sabaratnam, MPH Sr. Health Project Coordinator Div. of Adolescent Medicine, University of Rochester (585) 273-4616 Premini_Sabaratnam@urmc.rochester.edu or Jonathan D. Klein, MD, MPH Associate Professor of Pediatrics and of Community & Preventive Medicine University of Rochester (585) 275-7760 Jonathan_Klein@urmc.rochester.edu

  36. Or visit... The University of Rochester, Division of Adolescent Medicine, Leadership Education in Adolescent Health website at www.urmc.rochester.edu/gchas/div/adol/leah/resources.htm or The ACT for Youth, Center of Excellence website at www.actforyouth.net

More Related