PVAAS Overview: Evaluating Growth, Projecting Performance

1. PVAAS Overview:Evaluating Growth,Projecting Performance PVAAS Statewide Core Team Fall 2008

2. What do you think you already know about PVAAS? • What do you want to find out about PVAAS today? Goal:PVAAS will be used by teachers and administrators for annual school evaluation and planning, grade/subject level evaluation and planning, and for student level planning.

3. PDE Provides The Data Tools…

4. “Getting Results!™” A Framework for Continuous School Improvement Planning The framework provides guiding questions for root cause analyses in the following data environments: PSSA and AYP inquiry PVAAS 4Sight Benchmark Data Local Assessment Data Sets PDE Provides The Framework…

5. Review of Literature While there is a relationship between achievement and demographic variables, there is NO relationship between growth and demographic factors, such as socioeconomic status and racial/ethnic background. --VAAS can remove the effects of factors not under the control of the school. (McCaffrey, Lockwood, Koretz & Hamilton, 2003; Ross, Wang, Sanders, Wright & Stringfield, 1999a; Wright, Horn & Sanders, 1997).

6. What Is PVAAS?

7. PVAASIS: • A statistical analysis of existing assessment data. • A realistic measurement of a school’s progress/growth over time. • Progress data to add to achievement data as schools are analyzing their work with students and making improvement plans.

8. PVAAS: • Is not another test. • Is not being used for teacher-level accountability in Pennsylvania. • Does not use teacher-level data elements. • Does not yield a metric on the effectiveness of a teacher.

9. Current Data Collection Approaches High Achievement Average Achievement Low Achievement Achievement is based on each student’s ending point.

10. PVAAS Adds Another Variable To Achievement…Progress Below Growth Standard At or Above Growth Standard Progress is based on each student’s starting point.

11. The Scatter Plot Coordinate System Vertical Axis PSSA Percent Proficient or Advanced Growth Standard Boundary AYP Percent Proficient Target Horizontal Axis PVAAS Growth Measure

12. Scatter Plot Quadrants and Their Meanings Quad 2 Adequate Achievement Negative Growth Quad 1 Adequate Achievement Positive Growth Quad 4 Below Adequate Achievement Positive Growth Quad 3 Below Adequate Achievement Negative Growth

13. The TWO PVAAS Methodologies Looking Forward/Planning… PVAAS Projection Reports For Individual Students and Cohorts of Students Looking Back/Evaluation… Value-added Growth Reports For Cohorts of Students Today NO methodology to consider teacher effectiveness.

14. PVAAS Fall 2008 Reporting • Value-added growth reports in • Mathematics (grades 4 through 8 and 11) • Reading (grades 4 through 8 and 11) • Science (only 4, 8, and 11) • Writing (only 5, 8, and 11) • Projections to basic, proficient, advanced • 3  4 or 5 • 4  5 or 6 • 5  6 or 7 • 6  7 or 8 • 7  8 • 8  11 Grade Last Tested

15. Student Projections Wouldn’t it be great to know the likelihood that a student will be proficient on a future PSSA?

16. Student Projection Methodology • A separate longitudinal modeling process focused on the performance scores of individual students. • The model uses all of the available data (both math and reading) to project future PSSA performances in Math, Reading, Science and Writing. • Projections can be calculated for an individual student or groups of students.

17. PVAAS Student Projection Report

18. PVAAS Projection Reports for Groups of Students by Subgroup and/or Projected Proficiency Level

19. Identified Students Meeting Search Requirements

20. Projection Summary Reports • What is it? • Report that summarizes the numbers and percents of students in likelihood ranges of performing at the proficient level on a future PSSA exam. • How might a school use this report? • Intervention Planning • Resource Allocation • Strategic Planning • School Improvement Planning • Cautions • This report provides another indicator about likelihood of future performance.

21. Grade Projection Summary Report

22. School Projection Summary Report

23. District Projection Summary Report

24. Value-Added Wouldn’t it be great to know if you made at least one year’s growth with students last tested in grades 4-8?

25. Value-added Measure ofCohort GrowthGrades 4 through 8 in Reading and Math • Comparison to a State Growth Standard • The Growth Standard specifies the minimal acceptable academic gain from grade to grade for a cohort of students. • The use of a Growth Standard creates the possibility that ALL schools can demonstrate appropriate growth.

26. The Assessment Dilemma True Student Achievement Any test is just a snapshot in time!

27. Four Important Issues Difference in scores in consecutive years. 1. Measuring Gain 2. Growth Standard Convert all scores to same scale – NCE scale 3. Comparing Scores from year to year Combine current observed scores with all previous scores in students’ histories. 4. Estimating Achievement

28. PVAAS Value-added CohortGrowth Descriptors – Grade 4 – 8

29. PVAAS School ReportGrades 4 through 8 Every grade in the school has met or exceeded a year’s worth of growth? (“Getting Results!”™, p. 12)

30. Your Turn! Interpret this report!

31. PVAAS School ReportHigh School Based on a predictive methodology • A predicted score is calculated or each student; • Mean of predicted scores is calculated for students in a cohort; • Mean of observed scores of students in the cohort; • School Effect is the difference between the Observed and the Predicted Scores.

32. 11th Grade Growth Descriptors

33. School Report

34. PVAAS Performance Diagnostic ReportThe Key is the RED Whisker!

35. What the whiskers tell us… Exceeded Growth Standard; More than One Year’s Growth Green Met the Growth Standard; Made One Year’s Growth Yellow Growth Standard (One Year’s Growth) Below Growth Standard; Less than One Year’s Growth Rose

36. Performance Diagnostic Report Every performance levels (below basic, basic, proficient, advance) in each grade has met or exceeded a year’s worth of growth? (“Getting Results!”™, p. 12)

38. Greater Detail – based on % of students

39. Click on Hot Keys for lists of students.

40. Student Lists from Performance Diagnostic Reports

42. Custom Diagnostic Report • What is it? • Procedure to examine subgroup growth patterns based on user-defined educational criteria for groups of 15 or more students. • How might a school use this report? • Explore effects of intervention programs, etc. • Explore effects of varied curricular and/or instructional experiences. • Cautions • This report does not infer any causal relationships between any educational variables and student growth. • This report should NOT be used to estimate teacher effect on learning.

44. Custom Diagnostic ReportSelection Screen

45. Custom Diagnostic Report

46. Turn to someone next to you… What are some examples (2 or 3) of how PVAAS Value-added and Performance Diagnostic Reports can be used in your school?

47. PA’s Intent for PVAAS • Another Tool for Continuous Improvement • Measure of Growth to Add to Achievement/Status Measures • Not to Be Used in Isolation • No Teacher-Level Analyses

48. Additional Support Account Management • E-mail pdepvaas@iu13.org for any assistance needed regarding your PVAAS account or access. Professional Development • For assistance with professional development, interpretation and use of PVAAS reports, contact your local Intermediate Unit PVAAS contact. • If additional support is needed, your local IU will contact the PVAAS Statewide Core Team and/or the supporting PaTTAN PVAAS team.

49. Additional Professional Development • Each IU has a PVAAS contact • Fall Professional Development • 21 statewide webinars • 38+ days of hands-on professional development at IUs • Follow-up support • Professional Development Materials • PDE website

50. Questions:PVAAS Materials or Statewide Implementationpdepvaas@iu13.org717-606-1911PVAAS Report Web Sitehttps://pvaas.sas.com