2 Sessions March 9 and 10, 2011 Kristen Lewald, Ed.D. PVAAS Statewide Team for PDE

1. Pennsylvania Value-Added Assessment System (PVAAS):PVAAS Public Data ReleaseUpdate to PDE and PAIU 2 Sessions March 9 and 10, 2011 Kristen Lewald, Ed.D. PVAAS Statewide Team for PDE Lancaster-Lebanon IU 13

2. Today’s Session • PDE and PAIU Executive Director’s Meeting was held last Friday • Questions from public and districts/schools • Provide an update to PAIU • PDE joining us today as well

3. Agenda Achievement and Progress Available Data & Crosswalk Public Website Demonstration Myths and Concerns: Feedback Suggested Resources/Supports for Communicating PVAAS Questions

4. Achievement vs. Progress Achievement Progress Is the concept underlying value-added analysis and reporting Not correlated with demographic factors Dependent upon what happens as a result of schooling • The final result of an academic experience • Highly correlated with demographic factors, such as socioeconomic status • Affected by factors outside the school

5. Value-Added is… • A statistical analysis used to measure a district’s or school’s impact on the academic progress rates of groups of students from year to year. • Conceptually, a growth measure is approximately the difference between current achievement (current results) and prior achievement (prior results) with achievement being measured by an appropriate assessment, such as the PSSA. • However, PVAAS is NOT a simple comparison of two scores!

6. PVAAS Provides: Looking Forward/Planning… PVAAS Projection Reports For Individual Students and Cohorts of Students Public Site Looking Back/Evaluation… Value-added Growth Reports For Cohorts of Students Today

7. One of the inherent benefits of the PVAAS analyses is that all scores for all students are included in the analyses. Simplistic approaches are less precise and sometimes biased. While PVAAS does use individual student data to yield value-added reporting, there are NO individual student measures of growth yielded from PVAAS Key Concepts in Understanding PVAAS Reporting

8. Value-Added & PVAAS • Pennsylvania’s model for value-added is called PVAAS - the Pennsylvania Value-Added Assessment System. • PVAAS is based on the EVAAS Methodology – the Education Value-Added Assessment System. • The EVAAS methodology has been nationally reviewed and published.

9. PVAAS System • Uses EVAAS statistical methodology • EVAAS: Education Value-Added Assessment System • Mixed Model Multivariate Longitudinal Analyses • Lead Developer: Dr. Bill Sanders, Univ. of TN • Now: SAS, Inc.(Cary, NC) for EVAAS/PVAAS • Jim Goodnight, CEO

10. 8 Year History of PVAAS • Pilot from 2002-2005: 100 Districts • Fall 2006: Reporting Grades 4 and 6 Reporting to all 501 Districts • Fall 2007: Grades 4-8 Reporting to all Districts • Fall 2008: Grades 4-8, & 11 Reporting to all Districts • Fall 2009 and 2010: Full Reporting to 500 Districts • Math, Reading, Science, Writing • Grades 4-8, & 11 • Used as provision to meet AYP for NLCB – AYP Growth Model • February 2011: Release of Public Reporting Site https://pvaas.sas.com

11. PVAAS Data Provides Information to: Raise Achievement Close Achievement Gaps Decrease Dropouts Increase College Readiness

12. Key Concepts in Understanding PVAAS Reporting • PVAAS reporting reflects the effectiveness of your district’s or school’s Standards-Aligned System • PVAAS reporting reflects the district’s or school’s system regarding curriculum, assessment, and instruction

13. PVAAS Password-Protectedvs. Public Site PVAAS yields data on Districts, Schools, Grades, Subgroups, Students • Math, Reading, Science, Writing • Public • Protected

14. Achievement + Growth Achievement results (PSSA) and growth results (PVAAS) must be used together to get a complete picture of student learning. To view the achievement results of Pennsylvania's public districts/schools, go to: http://paayp.emetric.net/

15. Example: Achievement + Growth

16. Example: Achievement + Growth Same Schools

17. Overview of PVAAS Public Reports & Features: Screen Shots, THEN LIVE Web Demo

18. New PVAAS Login Page https://pvaas.sas.com

19. PVAAS Public Reports Use of Reports Tab to Select & View Reports

20. PVAAS Public Reports • Value-Added Summary Reports • District/LEA and School Level data only • Math and Reading • Grades 4-8 and 9-11 • School Search Capability • Allows users to find and view the progress of local schools, charter schools, and full-time CTCs across Pennsylvania. • Can search for similar schools based on grade levels tested, various demographics, Intermediate Unit (IU) region and/or county..

21. PVAAS Public Reports • Use of Tests Tab to View Reports at Different Grade Levels • Use of Subjects Tab to View Reports for Different Subjects

22. Reports on Public Site They have a different format from the district password protected site However, the measures on the PVAAS public site come directly from the reports on the district/school password protected site. We will cross walk between these two sites in this session.

23. 3 Key Resources for Public Reporting Guide to Public Reporting Crosswalk Two pager on public reporting

24. Purpose of District & SchoolValue-Added Data • Provides users with information to assist them in evaluating the overall effectiveness of a district/LEA or school on the academic progress of groups of students. • This report is NOT a report on teacher effectiveness!

25. Example of District Value-Added Summary ReportGrades 4-8, Math & Reading

26. Example of School Value-Added Summary ReportGrades 4-8, Math & Reading

27. Crosswalk: Where can I find this information on other reports? Public Site: School Value-Added Summary Report Restricted Site: School Value-Added Report

28. What is the Average Gain over Grades Relative to the Growth Standard? • Represents the average gain across the grade levels served between 4 and 8 compared to the Growth Standard. • It is the average academic growth of the district’s or school’s students, compared to the Growth Standard. • Answers the question, “How effective was the district/LEA/school in impacting the academic progress of its students compared to the Growth Standard?”

29. Example: Average Gain over Grades Relative to the Growth Standard • Average Gain over Grades Relative to the Growth Standard (on the public report) is the SAME as the Mean NCE Gain over Grades Relative to the Growth Standard on the School Value-Added Report (password-protected site)! 2.1 5th Grade Gain 0.2 4th Grade Gain 2 1.7 Average Gain over Grades Relative to Growth Standard

30. What is the Average Gain over Grades Relative to the State? • Represents the average gain across the grade levels served between 4 and 8 compared to the average progress of all students in Pennsylvania at the same grade levels. • It is the average academic growth of the district’s or school’s students, compared to the academic growth of students statewide. • Answers the question, “How much did the district/LEA/school impact the academic progress of its students compared to the progress of other students in Pennsylvania in those same grade levels?”

31. Example: Average Gain over Grades Relative to the State • Average Gain over Grades Relative to the State (on the public report) is the SAME as the Mean NCE Gain over Grades Relative to the State on the School Value-Added Report (password-protected site)! ? Average Gain over Grades Relative to Growth Standard for the State 1.7 Average Gain over Grades Relative to Growth Standard for the school 0.4 Average Gain over Grades Relative to the State

32. What is Average Gain over Grades Relative to the Growth Standard for the State? 3.0 Grade 4 State 3-Yr-Avg -0.4 Grade 5 State 3-Yr-Avg 2 1.3Average Gain over Grades Relative to Growth Standard for the State

33. Example: Average Gain over Grades Relative to the State • Average Gain over Grades Relative to the State (on the public report) is the SAME as the Mean NCE Gain over Grades Relative to the State on the School Value-Added Report (password-protected site)! 1.3 Average Gain over Grades Relative to Growth Standard for the State 1.7 Average Gain over Grades Relative to Growth Standard for the school 0.4 Average Gain over Grades Relative to the State

34. PVAAS Value-added Growth Descriptors Grades 4 – 8, Math and Reading

35. Example of District Value-Added Summary ReportGrades 9-11, Math & Reading

36. Example of School Value-Added Summary ReportGrades 9-11, Math & Reading

37. Crosswalk: Where can I find this information on other reports? Public Site: School Value-Added Summary Report Restricted Site: School Value-Added Report

38. PVAAS Value-added Growth Descriptors Grades 9-11, Math and Reading

39. What is the District/School Effect? • Provides an estimate of the district’s or school’s impact on students’ academic progress. Specifically, the District/School Effect is a function of the difference between the observed/actual PSSA achievement and the predicted PSSA achievement. • It is a measure of growth that the students tested in grade 11 have made over the past 3 years since being tested in grade 8 and uses data from grades 3-8. • If students score as expected (i.e., students’ observed scores are equal to their predicted scores), then the District/School Effect would be 0. A negative District/School Effect indicates students’ actual scores were lower than their predicted scores, while a positive District/School Effect indicates students’ actual scores were higher than their predicted scores. • Answers the question, “How effective was the district/LEA/school in promoting student academic growth and supporting students to meet or exceed their expected progress?”

40. Is it appropriate to compare the amount of progress made by a district/school to another district/school? • No! Not using the District and School Value-Added Summary Reports • Without taking the Standard Error into account, it is NOT possible to directly compare these gain values across districts/schools. • The color-coding of the growth measures (gain values) does in fact take into account the Standard Error. • Note the link to the color code legends on each report. • The Average Growth Index found on the School Search report feature takes the Standard Error into account and allows a more direct comparison across schools.

41. PVAAS Public Reports Use of Reports Tab to Select & View Reports

42. Purpose of School Search • Users can find and view the progress of public schools across Pennsylvania and search for similar schools based on grade levels tested, various demographics, Intermediate Unit (IU) region, and/or county.

43. School Search

44. School Search

45. Where can I find this information on other reports? Public Site: School Search Restricted Site: School Search

46. Which schools are included when I use School Search? • Schools with at least one tested grade in common as the “reference school” you selected AND any demographics selected, as well as IU or county region. • Example: Your reference school is a grade 6-8 school • Other schools included in the search may include grade 6-7 schools, grade 7-8 schools, K-6 schools, etc.

47. How are schools compared? • PVAAS: Average Growth Index • What is an Index? • A numerical scale used to compare variables with one another or with some reference number • Analogy: Consumer Price Index • A measure of the average change over a period of time • Statistical Indicator • Reflects patterns • The PVAAS Average Growth Index allows viewers to compare growth across schools.

48. What is the Average Growth Index? • A measure of student progress across the tested grade levels in a school. • This index is a value based on the average growth across grade levels and its relationship to the standard error so that comparison among schools is meaningful. If the standard error is not accounted for, users might get a skewed picture of the relative effectiveness of different schools. • For grades 4 through 8, the Average Growth Index is calculated by dividing the Average Gain over Grades Relative to the Growth Standard by the corresponding Standard Error. • For grades 9 through 11, the Average Growth Index is calculated by dividing the School Effect by the corresponding Standard Error.

49. Example: Average Growth Index, Grades 4-8 Average Growth Index = Average Gain over Grades Relative to the Growth Standard divided by the Standard Error

50. Example: Average Growth Index, Grades 4-8 Average Growth Index = Average Gain over Grades Relative to the Growth Standard divided by the Standard Error = 2.9 / 0.4 = 7.3 (due to rounding this may NOT be exactly what is reported on public site)