1 / 81

2011 STAR Interpreting and Using Results

2011 STAR Interpreting and Using Results. August 10, 2011 Webcast Webcast starts at 9:00 a.m. Objectives. Workshop participants will be able to: Describe the purposes of STAR reports Interpret STAR results Explain key statistics Compare and contrast types of reports

Ava
Download Presentation

2011 STAR Interpreting and Using Results

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 2011 STARInterpreting and Using Results August 10, 2011 Webcast Webcast starts at 9:00 a.m.

  2. Objectives Workshop participants will be able to: • Describe the purposes of STAR reports • Interpret STAR results • Explain key statistics • Compare and contrast types of reports • Identify proper uses of reports Post-Test Workshop

  3. Agenda • What’s New? • Results and Statistical Analysis • Using Results • Summary and Internet Reports • Data CDs • Individual Student Reports • Early Assessment Program Post-Test Workshop

  4. California Modified Assessment (CMA) reports Scale score, performance levels for ELA grade 9 Life Science, grade 10 Algebra I Percent correct for ELA, grades 10 and 11 Geometry Writing component of grade 4 CST and CMA ELA restored What’s New in 2011 Manual (M) 2 Post-Test Workshop

  5. Quiz Question 1 Which of these tests had scale scores reported for the first time in 2011? • CST for World History • CAPA for Science • CMA for Algebra I • CMA for Geometry Post-Test Workshop

  6. Quiz Question 1 Which of these tests had scale scores reported for the first time in 2011? C. CMA for Algebra I Post-Test Workshop

  7. Results: Purposes of STAR Reports M 3 • Report progress toward proficiency on the state’s academic content standards • Notify where improvement needed • To help students’ achievement • To improve educational programs • Provide data for state and federal accountability programs Post-Test Workshop

  8. Results: Performance Levels M 8−9 • State goal: All students score at proficient or higher • 350 or higher scale score • All CST • CMA for ELA in grades 3–9, math in grades 3–7 and Algebra I, and science in grades 5, 8, and 10 • STS for RLA and math in grades 2–7, excluding Algebra I • CAPA proficient: 35 or higher scale score Post-Test Workshop

  9. Results: Other Performance Levels M 8−9;Appendix B • Advanced • Basic cut score • CST: 300 • CMA: 300 for ELA in grades 3–9, math in grades 3–7, Algebra I, science in grades 5, 8, and 10 • STS: 300 for RLA and math in grades 2–7 (excluding Algebra I) • CAPA: 30 • Below basic • Far below basic • For each testing program, cut pointsvary for advanced and below basic by • Subject • Grade Post-Test Workshop

  10. Results: Scale Scores M 7−11 • Scale scores allow the same score to mean the same thing across test versions within grade and content area • Accounts for differences in difficulty • Scale score ranges by program: • CST, CMA (ELA grades 3–9, mathematics 3–7, Algebra I, science in grades 5, 8, and 10), STS (grades 2–7, excluding Algebra I): 150–600 for each grade and subject • CAPA: 15–60 for each level and subject Post-Test Workshop

  11. Results: Equating M 7 • Psychometric procedure • Adjusts for test difficulty • Additional information in the technical report on the CDE Web site Post-Test Workshop

  12. Results: Reporting Clusters(Content Areas) M 9−11;Appendix A • Three to six clusters for each subject • May be useful as indicators of individual or group strengths and weaknesses • But . . . Reporting clusters should be interpreted with caution Post-Test Workshop

  13. Results: Cluster Percent Correct Available for 2011 M 9−11;Appendix A • CST: all • CMA: • Grades 3–9 ELA, grades 3–7 math, Algebra I, grades 5, 8, and 10 science • STS: all Post-Test Workshop

  14. Results: Reporting ClusterCautions M 9−11;Appendix A • Based on small numbers of items; therefore, may not be reliable or generalizable • NOT equated from year to year • Should not compare reporting cluster percent correct from year to year Post-Test Workshop

  15. Interpreting Reporting Clusters or Content Areas in the Same Year M 8−12;Appendix A • Compare to percent-correct range of proficient students statewide Post-Test Workshop

  16. 2011 CST Reporting Clusters: Number of Questions and Average Percent Correct M Appendix A 2011 Post-Test Guide, Appendices A and C as posted on startest.org will be finalized with complete data on August 15. Post-Test Workshop

  17. Examples—Interpreting Reporting Clusters for the CST for Geometry M 11 Post-Test Workshop

  18. Quiz Question 2 What is a scale score? • Percent correct of all questions • Mean percent correct of all questions • An adjustment of this year’s and last year’s raw scores to show changes • An adjustment of the raw score to account for differences in difficulty Post-Test Workshop

  19. Quiz Question 2 What is a scale score? D. An adjustment of the raw score to account for differences in difficulty Post-Test Workshop

  20. Using Results M 3 • For instructional decisions with other data • Used in Academic Performance Index (API) calculations: CSTs, CAPA, CMA (only grades 3–9, Algebra I, Life Science) • Used in adequate yearly progress (AYP) calculations, ELA and mathematics: • Grades 2–8, CSTs • Grades 3–8 and Algebra I, CMA • Grades 2–8 and 10, CAPA Post-Test Workshop

  21. Year-to-Year ComparisonsDo Compare CSTs:Same Grade and Same Content Area M 12−15 • Mean scale score • Same content and grade, varying years • Percent in each performance level • Same content by grade across years • e.g., 2010 grade 10 ELA with 2011 grade 10 ELA Post-Test Workshop

  22. Year-to-Year ComparisonsDo Compare CSTs:Percent Proficient and Advanced M 12−15 • Percentage of students scoring at PROFICIENT and above • For a given grade and subject, e.g., Percent proficient and above for grade 3 math in 2010 and 2011 • For a given subject and aggregated grades, e.g., Percent proficient and above for grades 2– 6 mathematics in 2010 and 2011 • Across grades and a subject, e.g., Percent proficient and above in all courses and all grades Post-Test Workshop

  23. Year-to-Year ComparisonsDON’T Compare M 12−15 • Individual scale scores or statistics based on scale scores for different grades or content areas • Subjects by grade are independently scaled • Different content standards are measured in different grades • Cohorts across grades • Across tests • CMA grades 9−11, STS grades 8−11 to previous years • CMA grades 6−8, STS grades 5−7, only compare 2011 to 2010 • CAPA, CMA grades 3−5, STS grades 2−4, DON’T compare to years before 2009 Post-Test Workshop

  24. Example—Using CST Results to Compare Grade Results from Year to Year M 14 Post-Test Workshop

  25. Quiz Question 3 What is the best comparison for CST scores of students within a middle school? • 2010 mean scale scores for ELA of a cohort of grade 7 students with 2011 scale scores for ELA of the same students in grade 8 • 2010 mean scale scores for ELA for grade 8 students with 2011 mean scale scores for ELA for grade 8 students • 2010 mean percent correct scores for ELA with 2011 percent correct scores for ELA for the same students in grade 8 • 2010 mean percent correct scores for ELA for grade 8 students with 2011 mean percent correct for ELA for grade 8 students Post-Test Workshop

  26. Quiz Question 3 What is the best comparison for CST scores of students within a middle school? B. 2010 mean scale scores for ELA for grade 8 students with 2011 mean scale scores for ELA for grade 8 students Post-Test Workshop

  27. Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare . . . • To proficient students statewide • One cluster to another, same year • The same cluster to the same cluster, different years • To the average percent correct of all students in a class Post-Test Workshop

  28. Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare… • To proficient students statewide Post-Test Workshop

  29. Aggregate (Summary) Reports M 18−22 • What are they? • Student Master List Summary • Student Master List Summary End-of-Course (EOC) • Subgroup Summary • Back of reports: guide to abbreviations, score codes • Report Emphasis: CSTs • Criterion-referenced tests • Progress is measured in percent of students scoring proficient and advanced Post-Test Workshop

  30. Student Master List Summary M 19–20, M 29−35 • By grade or EOC subject • CSTs, CMA, CAPA, and STS • Lists subjects or EOC grades • % and # at each performance level • Mean scale score • CST • CAPA • CMA ELA in grades 3–9, math in grades 3–7, Algebra I, and science in grades 5, 8, and 10 • STS RLA and math in grades 2–7 [not Algebra I]) Post-Test Workshop

  31. Student Master List Summary (cont.) • Reporting cluster: mean percent correct • CST • CMA ELA in grades 3–9, math in grades 3–7, Algebra I, and science in grades 5, 8, and 10 • STS Post-Test Workshop

  32. Student Master List Summary M 35 Grade 7 Example Post-Test Workshop

  33. Student Master List SummaryBasic Statistics M 29−35 Post-Test Workshop

  34. Who Counts? Number Enrolled M 31 • Total CST/CMA and CAPA multiple-choice answer documents submitted as scorable • Minus • Documents marked as “Student enrolled after the first day of testing and was given this test” Post-Test Workshop

  35. Who Counts? Number Tested M 31 • All CST, CMA, CAPA, STS answer documents with one or more answers • Plus • Z = Tested but marked no answers • Not included • A = Students absent • E = Not tested due to significant medical emergency • P = Parent/guardian exemptions • T = Enrolled first day, not tested, tested at previous school • Students with inconsistent grades • Non–English learners who took the STS Post-Test Workshop

  36. Who Counts? Number and Percent Valid Scores M 31 • Number Valid Scores • For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct) • Percent Valid Scores • For the subject, number of valid scores divided by the number of students tested • Not included: • Incomplete tests • Modified tests • Non–English learners who took the STS • Unknown EOC mathematics (except grade 7 mathematics) or science tests • Inconsistent grades Post-Test Workshop

  37. Who Counts? Number Tested with Scores M 31 • All tests taken, including those taken with modifications, that receive a score • Not included: • Incomplete tests • Non–English learners who took the STS • Unknown EOC mathematics (except grade 7 mathematics) or science tests • Inconsistent grades Post-Test Workshop

  38. Student Master List SummaryPerformance Levels M 32 Post-Test Workshop

  39. Who Counts? Performance Levels M 32 • All CSTs, CAPA • CMA grades 3–9 ELA, grades 2–7 math, Algebra I, and grades 5, 8, and 10 science • STS grades 2–7 RLA and math (excluding Algebra I) • Advanced, proficient, basic, below basic • All valid scores falling in the performance level • Far below basic • All valid scores falling in the performance level • CSTs taken with modifications (in aggregate reporting and accountability only) Post-Test Workshop

  40. Who Counts?Mean Scale Scores M 32 • Average of valid scale scores • Can be used to compare results for the same content/grade across years Post-Test Workshop

  41. Student Master List Summary:Reporting Clusters M 32 Compare to: Average percent correct range for students statewide who scored proficient on the total test(See the Post-Test Guide, Appendix A.) Post-Test Workshop

  42. B = Blank C = Copied prompt I = Illegible L = Language other than English R = Refusal T = Off Topic W = Wrong prompt (prompt from an earlier administration) Student Master List Summary:Writing M 33 Post-Test Workshop

  43. Subgroup Summary:CSTs, CMA, CAPA, and STS M 42−58 • Disability status • Based on disability status for CST, CMA, STS • CAPA: each disability type • Economic status • Based on NSLP eligibility or parent education level • Gender • English proficiency • Ethnicity • Ethnicity for Economic Status (only for CSTs; CMA in grades 3–9 ELA; grades 3–7 math, Algebra I, and grades 5, 8, and 10 science; and CAPA) Post-Test Workshop

  44. Subgroup Summary: Ethnicity for Economic Status M 54 −58 Ethnicity for Economically Disadvantaged CST Grade 6 Example Post-Test Workshop

  45. Subgroup Summary: Ethnicity for Economic Status M 54 −58 Example: Economically disadvantaged for each ethnicity Post-Test Workshop

  46. Subgroup Summary: Ethnicity for Economic Status M 54-58 Post-Test Workshop

  47. Break — 10 minutes

  48. Internet Reports M 101−111 • Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS • Available to the public online for school, district, county, and state • “Students with Scores” = number tested with scores • CST Summaries of % Advanced and Proficient • More subgroups than paper reports • Parent education • Special program participation • Access from http://star.cde.ca.gov/ Post-Test Workshop

  49. Internet Demonstration Post-Test Workshop

  50. Internet Reports:CST Sample M 102 Post-Test Workshop

More Related