2008 STAR Interpreting and Using Results

1. 2008 STARInterpreting and Using Results August/September 2008

2. Objectives Workshop participants will be able to: • Describe the purposes of STAR reports • Interpret STAR results • Explain key statistics • Compare and contrast types of reports • Identify proper uses of reports Post-Test Workshop

3. Agenda • Quiz • Post Test Guide • What’s New? • Results and Statistical Analysis • Using Results • Summary and Internet Reports • Individual Student Reports • Teacher Reports Post-Test Workshop

4. Part I: General Info New in 2008 Interpreting Reports Comparing Results Part II: Report Descriptions Summary Individual Teacher Internet Part III: Appendices Reporting clusters Scale score ranges Statistical tables Test Score Verification procedures Demographic Data Correction procedures California reading list Glossary Post-Test Guide Layout Post-Test Guide Pages (Pp) i–iii Post-Test Workshop

5. CMA reports Percent correct Grades 3–5 STS reports Percent correct Grades 5–7 added CAPA Science percent correct Ethnicity for Economic Status subgroup summary (CSTs only) STAR Student Reports sorted for district All without addresses All with addresses What’s New in 2008 Post-Test Guide P 2 Post-Test Workshop

6. Results: Purposes of STAR Reports Post-Test Guide P 3 • Report progress of students toward proficiency on the state’s academic content standards • Inform where improvement needed • To help students’ achievement • To improve educational programs • Provide data for state and federal accountability programs Post-Test Workshop

7. Results: Performance Levels Post-Test Guide P 7 • State goal: PROFICIENT or higher • Scale score of 350 or higher for CSTs • Scale score of 35 or higher for CAPA • Other performance levels: • Advanced • Basic (300 CSTs, 30 CAPA) • Below Basic • Far Below Basic • CST and CAPA cut points vary for Advanced, Below Basic, and Far Below Basic by • Subject • Grade Post-Test Workshop

8. Results: Scale Scores Post-Test Guide P 7–8 • Scale scores allow same score to mean same thing across test forms within grade and content area and across time • Scale score ranges by program: • CSTs: 150–600 for each grade and subject • CAT/6 Survey: 1–999 for each grade and subject • CAPA: 15–60 for each grade and subject • CMA, STS, CAPA Science: Not available until 2009 Post-Test Workshop

9. Results: Equating Post-Test Guide Page 7 • Psychometric procedure • Adjusts for test difficulty • Additional information in CST Technical Report on CDE Web site • For 2008 • CSTs and CAPA (except science) are equated • CMA (3–5),CAPA (science), and STS (2–4) and will be equated starting in 2009 • CAT/6 nationally norm-referenced; not equated for STAR Post-Test Workshop

10. Results: Reporting Clusters(Content Areas) Post-Test Guide P 36 Appendices A & B • Four to six clusters for each content area (subject) • May be useful as indicators of individual or group strengths and weaknesses • But… Reporting clusters should be interpreted with caution Post-Test Workshop

11. Results: Reporting ClusterCautions Post-Test Guide P 8–10 Appendices A & B • Based on small numbers of items; therefore may not be reliable or generalizable • NOT equated from year to year • Should not compare reporting cluster percent correct from year to year Post-Test Workshop

12. Interpreting Reporting Clusters or Content Areas in the Same Year Post-Test Guide P 8–10; 84 Appendices A & B • CSTs—Performance on the clusters for students statewide scoring from • Lowest score for proficient to • Lowest score for advanced Post-Test Workshop

13. CSTs 2008 Reporting Clusters — Number of Questions and Average Percent Correct Post-Test Guide Page 138(Appendix A) Post-Test Workshop

14. Examples—Interpreting Reporting Clusters for CST for Geometry Post-Test Guide P 9 Post-Test Workshop

15. Using Results Post-Test Guide P 3 • For instructional decisions withother data: CSTs, CMA, CAPA, STS • Used in Academic Performance Index (API) calculations: CSTs, CAT/6 Survey,CAPA • Used in adequate yearly progress (AYP) calculations for grades 2–8 in ELA and math for CSTs and 2–8 and 10 in ELA and math for CAPA Post-Test Workshop

16. Year-to-Year ComparisonsDo Compare Same Grade and Same Content Area Post-Test Guide Pp 11–14 • Mean scale score • Same content and grade, varying years • Percent in each performance level • Same content by grade across years • e.g., 2007 ELA grade 10 with 2008 ELA grade 10 Post-Test Workshop

17. Year-to-Year ComparisonsDo Compare Same Grade or Same Content Area Post-Test Guide Pp 11–14 • Percentage of students scoring at PROFICIENT and above • For a given grade and subject, e.g., Percent proficient and above for grade 3 math in 2007 & 2008 • For a given subject and aggregated grades, e.g., % proficient and above for grades 2 to 6 math in 2007 & 2008 • Across grades and a subject, e.g., % proficient and above in all courses and all grades Post-Test Workshop

18. Year-to-Year ComparisonsDON’T Compare Post-Test Guide Pp 11–14 • Individual scale scores or statistics based on scale scores for different grades or content areas • Subjects by grade are independently scaled • Different content standards are measured in different grades • Cohorts across grades • CMA, STS scores because notscaled Post-Test Workshop

19. Example—Using CST Results to Compare Grade Results from Year to Year Post-Test Guide P 12 Post-Test Workshop

20. Aggregate (Summary) Reports Post-Test Guide Pp 17–21 • What are they? • Student Master List Summary • Student Master List Summary EOC • Subgroup Summary • Group Summary: CAT/6 Survey • Report Emphasis: California Standards Tests (CSTs) • Criterion-referenced tests • Progress is measured in percent of students scoring proficient and advanced Post-Test Workshop

21. Student Master List Summary Post-Test Guide Pp 17; 26–37 • By grade • CSTs, CMA, CAPA, and STS • Lists subjects • % and # at each Performance Level • Not for CMA, CAPA Science, STS • Mean Scale Score • Not for CMA, CAPA Science, STS • Reporting Cluster: Mean Percent Correct • Not for CMA, CAPA, and STS grades 5–7 Post-Test Workshop

22. Student Master List Summary Post-Test Guide P 27 Post-Test Workshop

23. Student Master List Summary Post-Test Guide P 27 Post-Test Workshop

24. Who Counts? Post-Test Guide P 28 • Number Enrolled • All CST, CMA, CAPA, and STS multiple-choice answer documents submitted as scorable • Minus • Documents marked as “Student enrolled after the first day of testing and was given this test” Post-Test Workshop

25. Who Counts? Post-Test Guide P 28 • Number, Percent Tested • All CST, CMA, CAPA, STS answer documents with one or more answers • Plus • Z = Tested but marked no answers • Not included • P = Parent/guardian exemptions • A = Students absent • T = Enrolled first day, not tested, tested at previous school • E = Not tested due to significant medical emergency • Students with inconsistent grades Post-Test Workshop

26. Who Counts? Post-Test Guide P 27 • Number Valid Scores • For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct). • Not included: • Incomplete tests • Modified tests • Non-English learners who took the STS • Unknown EOC math or science tests • Inconsistent grades Why does STS not have 100% valid scores? Post-Test Workshop

27. Who Counts? Post-Test Guide P 28 • Number Tested with Scores • All tests taken, including those taken with modifications, that receive a score • All STS and CMA tests taken • Not included • Incomplete tests • Non-English learners who took the STS • Unknown EOC math or science tests • Inconsistent grades Post-Test Workshop

28. Who Counts? Post-Test Guide P 29 • Performance Level Counts for CSTs • Advanced, Proficient, Basic, Below Basic • All valid scores falling in the performance level • Far Below Basic • All valid scores falling in the performance level • Modified tests (in aggregate reporting and accountability only) Post-Test Workshop

29. Student Master List SummaryPerformance Levels Post-Test Guide P 30 Post-Test Workshop

30. Student Master List SummaryReporting Clusters Post-Test Guide P 30 Compare to: Average percent correct range for students statewide who scored proficient on the total test Post-Test Workshop

31. B = Blank C = Copied Prompt I = Illegible L = Language other than English R = Refusal T = Off Topic W = Wrong prompt (Prompt from an earlier administration) Student Master List Summary:Writing Post-Test Guide P 30 Post-Test Workshop

32. Student Master List Summary: End-of-Course Post-Test Guide Pp 32–37 • By Subject • Lists each grade eligible to take test • Math (Grades 7–11) • General Math, Algebra I, Geometry, etc. • Science (Grades 9–11) • Earth Science, Biology, Chemistry, etc. • History–Social Science (Grades 9–11) • World History • Same statistics as grade-level Student Master List Summary Post-Test Workshop

33. Subgroup Summary:CSTs, CMA, CAPA, and STS Post-Test Guide Pp 38–64 • Disability Status • Based on “Disability Code” • Economic Status • Based on “NSLP” & “Parent Ed Level” • Gender • English Proficiency • Primary Ethnicity • Ethnicity for Economic Status: CSTs and CAPA (ELA and Math) only Post-Test Workshop

34. Subgroup Summary: Ethnicity for Economic Status Post-Test Guide Pp 60–64 Post-Test Workshop

35. Subgroup Summary: Ethnicity for Economic Status Post-Test Guide Pp 60–64 Post-Test Workshop

36. Subgroup Summary: Ethnicity for Economically Disadvantaged Post-Test Guide Pages 60–64 Post-Test Workshop

37. Internet Reports Post-Test Guide Pp 123–134 • Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS • Available to public online • More subgroups than paper reports • English-language fluency • Parent education • Special Program Participation • Access from http://star.cde.ca.gov Post-Test Workshop

38. CSTs Mean scale scores Percents by performance levels CST Summary Percent proficient and advanced CMA Average percent correct Grades 3–5 only STS Average percent correct Grades 2–7 only CAPA Mean scale scores Percents by performance levels Average percent correct for science CAT/6 Survey Mean scale scores NPR for “Avg. Student” Percent above percentile cuts Grades 3 and 7 only Internet Reports:Available Reports Post-Test Guide Pp 123-134 Post-Test Workshop

39. Internet ReportsAccess Reports Post-Test Guide Pp 124 Post-Test Workshop

40. Internet Reports:CST Sample Post-Test Guide P 127 Post-Test Workshop

41. Internet Reports:CST Summary Sample Post-Test Guide P 128 Post-Test Workshop

42. Internet Reports:CMA% Correct Post-Test Guide P 129 Post-Test Workshop

43. Internet Reports:STS% Correct Post-Test Guide Pp 133 Post-Test Workshop

44. Exercise • Work with a partner or in small groups • Plan what data you’ll present to leaders in your district • Remember the Do’s and Don’ts and Cautions Post-Test Workshop

45. Break — 15 minutes

46. Individual Reports Post-Test Guide Pp 70–112 • STAR Student Record Label • Adhesive label to affix to student’s permanent school record • STAR Student Master List • Alphabetical list of students and their scores • Tests listed in order within grade • CSTs-CAT/6 Survey • CMA-CAT/6 Survey • CAPA • STS • STAR Student Report: individual’s scores • 2 two-sided color copies • For parents/guardians • For each test Post-Test Workshop

47. Student Record Label:Student Name and Identification Post-Test Guide Pp 70–73 Post-Test Workshop

48. Student Record Label:CST for Grade 10 Results Post-Test Guide Pp 70–73 Post-Test Workshop

49. Student Master List:CSTs/CMA-CAT/6 for Grade 3 Sample Post-Test Guide Pp 74–79 Post-Test Workshop

50. Student Master List:CSTs/CMA-CAT/6 for Grade 3 Sample Post-Test Guide Pp 74–79 Post-Test Workshop