Youth PQA Basics

1. Youth PQA Basics Center For Youth Program Quality Youth Program Quality Assessment 1

2. Agenda • Welcome • Opening: Intro to the Youth PQA • Observational Note Taking • Fitting & Scoring • Building Your Pyramid • Self Assessment Q&A

3. The Quality Counts Initiative Why Quality Counts, Who We Are, & Where We’re Going

4. WHY? Because Rhetoric and Reality Don’t Match The American Dream All youth ready, every family and community supportive, each leader effective. The American Reality Only 4 in 10 youth ready, only 1 in 3 youth supported, too few leaders effective. The American Dilemma Fragmentation, complacency, and low expectations of youth, communities and leaders The Ready by 21™ Challenge Change the odds for youth by changing the way we do business 4

5. Ready for Work Youth Employment Outcomes …There is Increasing Agreement on Skills Needed for the 21st Century… Ready for College Academic Outcomes Specific Vocational Knowledge & Skills Subject Matter Knowledge 21st Century Skills & Content Information & Media Literacy * Creativity * Intellectual Curiosity * Critical & Systems Thinking * Accountability and Adaptability * Communication * Problem Solving * Interpersonal Skills * Social Responsibility * Financial Literacy * Global Awareness * Civic Literacy * Self-Direction Cultural, Physical & Behavioral Health Knowledge & Skills Ready for Life Youth Development Outcomes

6. Researchers Agree on What It Takes to Support Development The National Research Council & Institute for Medicine list the following key features of positive youth development settings: Physical and psychological safety Appropriate structure Supportive relationships Opportunities to belong Positive social norms Support for efficacy and mattering Opportunities for skill-building Integrationof family, school and community efforts - Community Programs to Promote Youth Development, 2002 6

7. Longitudinal Studies Confirm that these Supports Make a Difference 7

8. Providing these Supports Can Change the Odds from 4 in 10 doing well to 7 in 10 doing well* . Gambone/Connell’s research suggests that if all young people got the supports they needed in early adolescence, the picture could change… 8

9. `` ` l Systems for QualityAccountability Policies in Places • YPQA is part of state and county accountability policies: • Cross sector (DHS& DOE) snapshots: Iowa, Washington, Arkansas • Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico, • Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach Rochester Grand Rapids Washington* M i n n e a p o l i s Chicago New York e t r o i t Iowa Indianapolis Rhode Island m b u s Georgetown Divide Columbus St. Louis Kentucky Oklahoma Nashville Austin West Palm Beach County

10. WHO WE ARE - 3 Nationals, 5+ States, 7 Localities STATES (w/ participating localities) Iowa Linn County, N. Central Iowa, Polk County/Des Moines Kentucky Lexington, Louisville New York Broome, Onondaga, Orange & Rockland Counties Oklahoma Norman, Tulsa Rhode Island Central Falls, Newport, Pawtucket Washington State (Honorary QC member) • LOCALITIES • Austin, TX • Georgetown Divide (Black Oak Mine), CA • Columbus (Bartholomew County), IN • Grand Rapids, MI • Indianapolis, IN • Nashville, TN • St. Louis, MO • NATIONAL PARTNERS • The Forum for Youth Investment • The Center For Youth Program Quality (formerly High/Scope Youth Development Group) • AED National Training Institute for Community Youth Work 10

11. WHERE WE’RE GOING - Four Domains Capacityto Assess & Improve Programs Strong Policy / Leadership Horsepower A Strong&Stable Program Base Capacity to Recruit, Train, Retain Workforce The Quality Counts initiative is designed to help you maximize your capacity to improve the quality and reach of youth programs by taking concrete steps in four areas (domains) to ensure that you have:. 11

12. Why bother paying attention to youth program quality? Network leaders build awareness and communication Programs see their strengths and areas to improve, receive critical feedback, plus improve cmty networking and collaboration Funders gain an important metric Youth can have better program experiences 12

13. What is program quality? ??? inputs inputs outcomes inputs youth program Another way to say it: What do we want to see in high quality youth programs? 13

14. High POS Quality Several ways to organize: Readin’ + ‘Ritin + ‘Rithmatic (old-school) Affect + Active Learning + Metacognition (Education) Relationships + Relevance + Rigor (Education 2.0) Relatedness + Autonomy + Competence (Psychology SDT) Safety + Belonging + Esteem (Psychology old-school) 8 Features of Positive Youth Development Settings (PYD) Relationship + Task + Increasing Complexity Content Therapeutic process

15. The Pyramid of Program Quality Plan Make choices Engagement Reflect Lead and mentor Be in small groups Partner with adults Interaction Experience belonging Encouragement Reframing conflict Supportive Environment Skill building Session flow Active engagement Welcoming atmosphere Psychological and emotional safety Safe Environment Program space and furniture Emergency procedures Healthy food and drinks Physically safe environment Youth Voice and Governance Professional Learning Community

16. What is the Youth PQA? 1. A validated instrument designed to assess the quality of youth programs and identify staff training needs. 2. A set of items that measures youth access to key developmental experiences. 3. A tool which produces scores that can be used for comparison and assessment of progress over time.

17. The Youth PQA and Maslow Actualization B-Needs Esteem Needs Engagement Interaction(youth-youth; peer community) Love/Belonging Needs D-Needs Supportive Environment(adult-youth relationships) Safety Needs Safe Environment PQA Subscales (within pyramid) Maslow categories

18. Scores From Validation Studies

19. Youth PQA Domain ScoresN= 735 offerings (all unique staff) in 180 organizations

20. Contrasting Pedagogy ProfilesN=599 offerings in 120 organizations 5 = occurred for everyone 3 = occurred for some 1 = did not occur

21. SAE System Accountability Environment PLC Professional Learning Community POS Point Of Service How we think about DDCI- People change not programs Prochaska, J.O., & DiClemente, C.C. (1982). Transtheoretical therapy toward a more integrative model of change. Psychotherapy: Theory, Research and Practice, 19(3), 276-287. Maintenance Repeat cycle Action Implementation & coaching Preparation Planning with data Contemplation Quality assessment Individual Change Model Organizational Context The Center for Youth Program Quality

22. Designing Quality Improvement Systems (QIS) Mostly Mangers Managers and direct staff TOTs for quality assess, coaching, and youth work methods (f,g) SystemCapacity Use of on-line dashboards and training (d) External quality assessment (a) Quality Advisor (e) Coaching & Training External Quality Report with Norms Self-Assessed Quality Report Targeted youth work methods training for direct staff (h) Use of on-line dashboards and training (d) Quality coaching by managers (i) Program Staff Skill & Knowl Self-assessment of Quality (b) Planning with Data (c) Phase 2: Impact & Sustainability Phase 1: Readiness & Capacity

23. Defining the Purpose of Your QIS Lower Stakes Program Self-Assessment Rough data toget staff thinking anddiscussing programquality in the contextof best practice Less time Less money Impact internal audiences (the creative middle) Higher Stakes ExternalAssessment Precise data forinternal and externalaudiences for evaluation,monitoring, accountability,improvement, reporting More time More moneyImpact internal and external audiences

24. Columbus IndianaPhase 1: Building Local Capacity POSPoint Of Service SAESystem Accountability Environment PLCProfessional Learning Community STEP 2a Self-assessment STEP 1 Decide to build system STEP 3 Plan for improvement STEP 4 Carry out plan STEP 5 Measure change STEP 2b External assessment August 26 Youth PQA Basics January 27-28Ext Assessment October 8 Planning with Data Improvement Plan Annually Program SA Ext Assessment Observe-Reflection Planning with Data Opt Phase 2 Method Workshops Quality Coaching August 25 Quality Matters Presentation

25. Step 1: Pilot Sites Attend Training- Youth PQA Basics, External Assessor

26. Step 2: Pilots Select Self-Assessment Teams and Develop Schedule

27. Step 3: Self Assessment

28. Step 4: Submit Scores

29. High/Scope’s eTools Web Siteetools.highscope.org Your one stop shop for: • Online training • Intro to Youth PQA • Intro to Scores Reporter • Online assessment • Submitting Youth PQA Scores • Generate reports

30. Online PQA Scores Reporteretools.highscope.org • 3 levels of users • Youth Program Facilitators • Youth Program Directors • Network Users

31. Online PQA Scores ReporterEnter Data with mouse or keyboard

32. Online PQA Scores ReporterView & Print Reports Your programs scores can automatically be compared to national norms

33. Online PQA Scores ReporterView & Print Reports

34. Step 5: Teams attend Planning with Dataand create Program Improvement Plans

35. Step 6: Staff members work the plan! Repeat Youth PQA annually…

36. Does it Work? Findings from Several Samples • POS quality-outcomes findings: • Supportive environment related to: Attendance • Interaction related to: Interest in program • Engagement related to: Sense of challenge, sense of growth, school-day reading, school-day suspension • Note: No offerings get to high engagement without high support and high interaction • Quality Improvement (YPQI) Findings • Scores increase from pre to post • Scores increase in the targeted areas more • Management practices are related to quality change (Vision, Feedback, Continuity)

37. Questions about… Purpose? Process? Pilot? Next Steps…

38. Youth Program Quality Assessment

39. Structure of the Youth PQA Form BOrganizational Interview Ask questions, write, score (2 hours) Form AObservation Watch, write, score (3 hours) Program Offering 1 Program Offering 2 Organization Program Offering 3 Program Offering 4

40. Sample indicator “subscale” “item” “indicator row” The Youth PQA consists of 2 scales (Form A & Form B); 7 subscales (4 in A, 3 in B) 30 items (18 in A, 12 in B); 103 indicator rows (60 in A, 43 in B)

41. Indicator Scramble What are the key words? What are the key quantities? What are we counting?

42. Sample Anecdotes • Youth sort of identified with program offering. III-L (3rd row) • Kids sometimes taunt each other and try to annoy each other, but they seem to get along, too. I-A (1st row) • Kevin dominated committee meetings he sat in on. Did better job of listening and acknowledging comments in the large group. III-O (1st row)

43. Observational Note Taking

45. Follow-up questions • Only ask for items with question(s) in the evidence column. • Ask the questions as they are printed.

46. Fit and Score (label practice) • Try to place each label in the correct row for each item. Look over the anecdotal evidence for each item and see how the anecdotes are a “fit” with the indicator rows. • Score each indicator row. For each row, read through the levels of the indicator row and select the level - 1, 3, or 5 - that most closely agrees with your evidence. Pay close attention to words like some, most, and always. Start at level 5 to see the full description of necessary evidence and check agreement with the anecdotal evidence by looking to the lower levels.

47. More on Scoring the YPQA • Fitting data to the indicator rows can eventually happen in your head while you are observing. • Always try to see multiple items in every interaction and cross-reference constantly. • Look to a preponderance of evidence but favor higher scores. • If you lack evidence with “fit,” collect more data.

48. Assessing POS Quality using Youth PQALet’s Try It! Watch, Write, Score Collecting objective anecdotal evidence Scoring rubrics Reliability Instructions In this activity you will watch a short (4 minute) video clip, take notes, and score a few indicators based on the video. Before you watch the video, get ready: Make sure you have your hardcopy of Form A of the Youth PQA in front of you for this activity. Make sure you have scrap paper and a pen or pencil handy Review items II-J, III-M, and IV-Q (you will be scoring one indicator from each) While you watch the video take notes by hand. Make sure your notes are as objective as possible. Try to capture actual quotes. 48

49. More on Self-Assessment • Self-assessment only works if program (direct) staff are involved. • Try to get a good mix of program times and offerings. • Assessment is different from evaluation. Assessment gives you the opportunity to see where you are, whereas evaluation suggests judgment. • Remember, the conversation is the most important part of self-assessment.

50. Youth PQA Step by Step