Understanding CAASPP 2015 Reporting Workshop: Scoring and Interpreting Results

1. California Assessment of Student Performance and Progress (CAASPP) 2015 Post-Test Workshop: Reporting Summative Assessment Results

2. Agenda • Purpose • Principles of Scoring • Understanding the Reports • Using the Online Reporting System • Overview of the Reporting Timeline • Interpreting, Using, and Communicating Results

3. Purpose

4. Purpose • By the end of this workshop, viewers will know how to: • Understand the scoring process • Understand the components of each report • Use the Online Reporting System (ORS) to view partial and preliminary results • Become familiar with the reporting timeline • Interpret, use, and communicate results

5. Focus of this Workshop • Overview of the reporting system for all CAASPP operational summative assessments: • Smarter Balanced English language arts/literacy (ELA) and mathematics • California Standards Tests (CSTs), California Modified Assessment (CMA), California Alternate Performance Assessment (CAPA) for Science • Standards-based Tests in Spanish (STS) for Reading/Language Arts (RLA) • Focus on Smarter Balanced ELA and mathematics

6. Principles of Scoring

7. Goals of the Section • Provide an overview of computer adaptive testing (CAT) and scoring • Describe the relative contribution of the performance tasks (PTs) and the CAT to the overall scores • Describe the • Score scale • Achievement levels • Error Bands • Claims

8. Computer Adaptive Testing:Philosophy “Computer adaptive testing (CAT) holds the potential for more customized assessment with test questions that are tailored to the students’ ability levels, and identification of students’ skills and weaknesses using fewer questions and requiring less testing time.” Shorr, P. W. (2002, Spring). A look at tools for assessment and accountability. Administrator Magazine.

9. How Does a CAT Work? • Each student is administered a set of test questions that is appropriately challenging. • The student’s performance on the test questions determines if subsequent questions are harder or easier. • The test adapts to the student item-by-item and not in stages. • Fewer test questions are needed as compared to a fixed form to obtain precise estimates of students’ ability. • The test continues until the test content outlined in the blueprint is covered.

10. How Does a CAT Work?Example: A Student of Average Ability Expanded Very High Ability High Med-High Medium Med-Low Low Expanded VeryLow 1 2 3 4 5 6 7 8 9 10 Test Questions Answers (R/W) R R W R W W W W R R

11. Computer Adaptive Testing:Behind the Scenes • Requires a large pool of test questions statistically calibrated on a common scale with ability estimates, e.g., from the Field Test • Uses an algorithm to select questions based on a student’s responses, to score responses, and to iteratively estimate the student’s performance • Final scale scores are based on item pattern scoring

12. Computer Adaptive Testing:Practical Considerations • Each student’s test is constrained to ensure coverage of the full range of appropriate grade level content, e.g., ELA test cannot consist of only Reading Informational items. • The exposure of test questions to many students is constrained to maintain test security. • Sets of test questions based on a common passage or stimulus constrains the ability to adapt within the set. • The responses must be machine-scored to select the next question. • Human-scored performance task responses are combined later with the CAT results.

13. Scoring the CAT • As a student progresses through the test, his or her pattern of responses is tracked and revised estimates of the student’s ability are calculated. • Successive test questions are selected to increase the precision about the level of achievement given the current estimate of his or her ability. • Resulting scores from the CAT portion of the test are based the specific test questions selected as a result of the student’s responses, but NOT the sum of the number answered correctly. • The test question pools for a particular grade level are designed to include enhanced pool of test questions that are more or less difficult for that grade but still matching the test blueprint for that grade.

14. Human Scored Items in the CAT • Some items administered on the Smarter Balanced adaptive test component require human scoring of items • The adaptive algorithm will select these items based on performance on prior items. • Since these items cannot be scored in real time by a human, performance on these items will not impact later item selection.

15. Performance Tasks (PTs) • In all Smarter Balanced tests, a PT and a set of stimuli on a given topic are administered as well as the CAT. • PTs are administered at the classroom/group level so they are not targeted to students’ specific ability level. • The items associated with the PTs may be scored by machine or by human raters.

16. Final Scoring • For each student, the responses from the PT and CAT portions are merged for final scoring. • Resulting ability estimates are based on the specific test questions that a student answered, not the total number of items answered correctly. • Higher ability estimates are associated with test takers who correctly answer difficult and more discriminating items. • Lower ability estimates are associated with test takers who correctly answer easier and less discriminating items. • Two students will have the same ability estimate if they have the same set of test questions with the same responses. • It is possible for students to have the same ability estimate through different response patterns • This type of scoring is called “Item Pattern Scoring.”

17. Final Scoring: Contribution of CAT and PT Sections

18. Final Scoring: Contribution of CAT and PT Sections (cont.) • Based on the test blueprint, the CAT section is emphasized because there are more CAT items/points than PT items/points. • Claims with more items/points are emphasized. • Mathematics: Concepts and Procedures  Problem Solving/Modeling and Data Analysis  Communicating Reasoning • ELA: Reading  Writing  Speaking/Listening  Research • Because scores are based on pattern scoring, groups of items that are more difficult and discriminating will have a larger contribution on final scores. • Therefore there is no specific weight associated with either PT or CAT Sections

19. Final Scoring: Mapping • After estimating the student’s overall ability, it is mapped onto the reporting scale through a linear transformation. • Mathematics: • Scaled Score = 2514.9 + 79.3 * • ELA: • Scaled Score = 2508.2 + 85.8 * • Limited by grade level lowest and highest obtainable scaled score

20. Properties of the Reporting Scale • Scores are on a vertical scale. • Expressed on a single continuum for a content area • Allows users to describe student growth over time across grade levels • Scale score range • ELA/Literacy: 2114–2795 • Mathematics: 2189–2862 • For each grade level and content area, there is a separate scale score range.

21. Smarter Balanced Scale Score Ranges by Grade Level

22. Achievement Levels • Achievement level classifications based on overall scores • Level 1—Standard Not Met • Level 2—Standard Nearly Met • Level 3—Standard Met • Level 4—Standard Exceeded

23. Achievement Levels by Grade

24. Smarter Balanced Scale Score Ranges for ELA/Literacy

25. Achievement Levels by Grade

26. Smarter Balanced Scale Score Ranges for Mathematics

27. Measurement Precision:Error Bands • For each scale score estimated for a student, there is measurement error associated with each score. An error band is a useful tool that describes the measurement error associated with a reported scale score. • Error bands are used to construct an interval estimate corresponding to a student’s true ability/proficiency for a particular content area with a certain level of confidence. • The error bands used to construct interval estimates were based on one standard error of measurement. • If the same test is given to student multiple times, about 68 percent of the time, the student will score within this band.

28. Achievement Levels for Claims • Achievement Levels for claims are very similar to subscores. They provide supplemental information regarding a student’s strengths or weaknesses. • No achievement level setting occurred for claims. • Only three achievement levels for claims were developed since there are fewer items within each claim. • Achievement levels for claims are based on the distance a student’s performance on the claim is from the Level 3 proficiency cut. • A student must complete all items within a claim to receive an estimate of his or performance on a claim.

29. Achievement Levels for Claims (2) • A student’s ability, along with the corresponding standard error, are estimated for each claim. • The student’s ability estimate for the claim is compared to the Level 3 proficiency cut . • Differences between and greater than 1.5 standard errors of the claim would indicate a strength or weakness.

30. Achievement Levels for Claims (3) At/Near Standard Below Standard

31. Achievement Levels for Claims (4) Above Standard

32. Understanding the Reports

33. Available Reports

34. Secure Reporting • * Access to ORS will be granted to CAASPP Test Site Coordinators in August. • † PDFs of the Student Score Reports will be available in TOMS. • †† LEAs must forward or mail the copy of the CAASPP Student Score Report to each student’s parent/guardian within 20 working days of its delivery to the LEA.

35. Preliminary Test Results: Student and Aggregate • Through the Online Reporting System (ORS) • Available approximately three to four weeks after student completes both parts—CAT and PT—of a content area • Added daily Use Caution: The results are partial and may not be a good representation of the school or district’s final aggregate results. The results are preliminary; the processing of appeals may result in score changes.

36. Student Score Reports (ISR): Overview • One-page report • Double-sided: • All Smarter Balanced • CAPA for Science • Single-sided: • CST/CMA for Science (Grade 10) • STS for RLA • Student’s final CAASPP test results • Reports progress toward the state’s academic content standards • Indicates areas of focus to: • Help students’ achievement • Improve educational programs • LEA distributes to parents/guardians

37. Student Score Reports: Shipments to LEAs • Two copies of each student’s Student Score Report • One for the parent/guardian • One for the school site • 2015 LEA CAASPP Reports Shipment Letter • 2015 School CAASPP Reports Shipment Letter Note: Per California Code of Regulations, Title 5, Section 863, LEAs must forward one copy to parent/guardian within 20 business days. Schools may file the copy they receive, or they may give it to the student’s current teacher or counselor. If the LEA receives the reports after the last day of instruction, the LEA must mail the pupil results to the parent or guardian at their last known address. If the report is non-deliverable, the LEA must make the report available to the parent or guardian during the next school year.

38. Test Results Reported on the Student Score Reports For students who took Smarter Balanced ELA and mathematics, CST, CMA or CAPA for Science For students who took STS RLA

39. Elements of the Student Score Report Back Page Front Page 5 1 2 3 6 7 4 8

40. Elements of the Student Score Report Front Page 1

41. Elements of the Student Score Report Front Page 2

42. Elements of the Student Score Report Front Page 3

43. Elements of the Student Score Report Front Page 4

44. Elements of the Student Score Report Back Page 5

45. Elements of the Student Score Report Back Page 6

46. Elements of the Student Score Report Back Page 7

47. Elements of the Student Score Report: ScienceGrades 5, 8, & 10 only Back Page 8

48. Elements of the Student Score Report: Early Assessment Program Grade 11 only Back Page 8

49. Student Score Reports (cont.) • A guide explaining the elements of student score reports will be available electronically on the caaspp.org reporting Web page.

50. Final Student Data File • Downloadable file in CSV format • Data layout to be released soon on caaspp.org • Includes test results for all students tested in the LEA • Available within four weeks after the end of an LEA’s test administration window in TOMS • Additional training planned