May–June 2017 Presented by Educational Testing Service (ETS) and WestEd

1. Post-Test Workshops: Using Assessment Results to Inform Teaching and LearningSession 1: Accessing Scores, Reports, Resources, and ToolsSession 2: Analyzing Summative Assessment Results to Inform Teaching and Learning May–June 2017 Presented by Educational Testing Service (ETS) and WestEd

2. Before We Get Started . . . Name: Passcode: • Internet • Materials • http:/tinyurl.com/lrpqlym • Question Cards • Housekeeping • Facilities • Parking • Refreshments • Evaluation • http://tinyurl.com/mv2royk 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

3. Agenda 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

4. “California’s testing system makes improving instruction a priority.” —Tom Torlakson, State Superintendent of Public Instruction 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning The purpose of the in-person workshop is to inform local educational agencies (LEAs) of the various components of the 2016–17 California Assessment of Student Performance and Progress (CAASPP) student results and reports and to discuss how this information might be used to inform teaching and learning.

5. Objectives Provide general information about the Smarter Balanced assessments in order to focus on the effective use of the 2016–17 summative assessment results Prepare LEA CAASPP coordinators to access the 2016–17 test results Provide supplementary resources and reference materials to assist LEA CAASPP coordinators with their local training efforts Provide an explanation of achievement levels, claims, and targets and how to interpret them 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

6. Setting the Stage

7. A Balanced Assessment System 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

8. The Smarter Balanced Hierarchy ofItem Development and Reporting of Scores Overall Claims Content Claims Targets Evidence Statements Items 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

9. Relationships Between Common Core Standards, California Frameworks, and Assessments English Language Arts and Literacy in History/Social Studies, Science, and Technical Subjects English Language Arts/Literacy Content Specifications (Claims) English Language Arts/English Language Development Framework English Language Arts/Literacy Item Specifications (Claims and targets per claim) English Language Arts/Literacy Summative Assessment English Language Arts/Literacy Assessment Blueprint Mathematics Content Specifications (Claims) Mathematics Item Specifications (Claims and targets per claim) Mathematics Framework for California Public Schools Mathematics Mathematics Summative Assessment Mathematics Assessment Blueprint 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

10. These tests are connected to what is taught in the classroom. Talk to your elbow partner: consider how you currently use the summative results to inform teaching and learning. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

11. The Process of Scoring

12. Data Flow California Longitudinal Pupil Achievement Data System (CALPADS) Test Operations Management System (TOMS): Test Assignments and Settings Student Demographic Data Test Settings and Basic Student Data Test Delivery System Student Responses Online Reporting System (ORS) Scoring System Test Results TOMS (Reporting) 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

13. CALPADS Reminders Confirm that your student demographic data are up-to-date in CALPADS and TOMS prior to the end of your LEA’s selected testing window. Students who will be returning to the same school the following year should not be re-enrolled in the next grade before the LEA’s selected testing window closes. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

14. Objectives of This Section • Provide an overview of computer adaptive testing (CAT) and scoring • Describe the relative contribution of the performance tasks (PTs) and CAT to the overall scores • Describe: • Score scale • Achievement levels • Claims • Assessment targets 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

15. How Does a CAT Work?Example: A Student of Average Ability Expanded Very High Ability High Med–High Medium Med–Low Low Expanded VeryLow 1 2 3 4 5 6 7 8 9 10 Test Questions Answers (right/wrong) R R W R W W W W R R 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

16. CAT: Practical Considerations • Each student’s test is constrained to ensure coverage of the full range of appropriate grade-level content (e.g., the ELA test cannot consist of only reading informational items). • The level of test-question exposure is constrained to maintain test security. • Sets of test questions based on a common passage or stimulus constrain the ability to adapt within the set. • The test question pools for a particular grade level are designed to include an enhanced pool of test questions that are more or less difficult for that grade but still match the grade’s test blueprint. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

17. CAT Scoring • As a student progresses through the test, his or her pattern of responses is tracked. • Revised estimates of the student’s ability are calculated. • Successive test questions are selected to increase the precision of the student’s level-of-achievement estimate, given the current understanding of his or her ability. • The adaptive test will select these items based on the student’s performance on prior items. • Scores from the CAT portion of the test are based on the specific test questions selected as a result of the student’s responses. • Scores are NOT based on the sum of the number of questions answered correctly. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

18. Human-Scored Items in the CAT Some items administered on the Smarter Balanced CAT component require human scoring. Since these items cannot be scored in real time by a human, the student’s performance on these items will not impact subsequent item selection. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

19. Performance Tasks (PTs) In all Smarter Balanced tests, a PT and a set of stimuli on a given topic are administered as well as the CAT. PTs are not adaptive and, therefore, are not targeted to a student’s specific ability level. The items associated with the PTs may be scored by a machine or by human raters. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

20. Final Score • For each student, the responses from the PT and CAT portions are merged for final scoring. • The resulting estimates of achievement are based on the specific test questions that a student answered and not the total number of items answered correctly. • Higherestimates of achievement are associated with test takers who correctly answer difficult and more discriminating items. • Lowerestimates of achievement are associated with test takers who correctly answer easier and less discriminating items. • It is possible for students to have the same estimate of achievementthrough different response patterns. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

21. Final Scoring: Contribution of the CAT and PT Sections 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

22. Final Scoring: Contribution of the CAT and PT Sections • Relative contribution of the PT and CAT sections on the total score will vary across students since the overall score • Is based on pattern scoring • Depends on performance on the CAT section • Depends on performance on the PT section • Depends on the average difficulty of the PT section • Ex: A PT might have a significant negative contribution on a students overall score if they did extremely well on the CAT but performed poorly on a very easy PT section • Upshot: All parts of the Smarter Balanced Assessments are important 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

23. Available Summative Assessment Scores 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning • Scale scores • Achievement levels • Not met, nearly met, met, exceeded • Claim levels • Below, near, above • Target scores • Worse, similar, better • Insufficient information

24. Properties of the Reporting Scale • Scores are on a vertical scale. • Expressed on a single continuum for a content area • Allows users to describe student growth over time across grade levels • Scale score ranges • ELA: 2114–2795 • Mathematics: 2189–2862 • For each grade level and content area, there is a separate scale score range. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

25. Smarter Balanced Scale Score Ranges by Grade Level and Content Area 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

26. Smarter Balanced Scale Score Ranges and Achievement Levels for ELA 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

27. Threshold Scale Scores by Grade 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

28. Smarter Balanced Scale Score Ranges and Achievement Levels for Mathematics 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

29. Threshold Scale Scores by Grade 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

30. Claim Areas English Language Arts/Literacy Reading Writing Listening Research/Inquiry Mathematics Concepts and Procedures Problem Solving/Modeling and Data Analysis Communicating Reasoning 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

31. Claim Scores 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning Reported at an individual and aggregate level Reported for all claims in ELA and mathematics

32. Achievement Levels for Claims Achievement levels for claims are very similar to subscores. They provide supplemental information regarding a student’s strengths or weaknesses. A student must complete all items within a claim to receive an estimate of his or her performance on a claim. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

33. Why are there only three achievement levels for Claims? Three achievement levels for claims were developed. Achievement levels for claims are based on the distance a student’s performance on the claim is from the Level 3 “standard met” criterion. 2016–17 Post-Test Workshops: Connecting Assessments to Instruction

34. Assessment Targets Assessment targets connect the content standards to evidence that will be collected from the assessment. Targets map the CCSS onto assessment evidence that is required to support the content categories and claims. Targets are used to guide the development of items and tasks that will measure the CCSS. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

35. Assessment Target Reports Assessment targets are group-level indicators (e.g., for a classroom). Target reports in the ORS indicate group-level strengths or areas for improvement for specific standards that are addressed in each claim in the ELA and Mathematics Concepts and Procedures claim. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

36. Assessment Target Reports Example: Mathematics Target Report Assessment Target Report Quick Start Guide available at http://www.caaspp.org/administration/reporting/index.html 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

37. Understanding the Reports

38. Available Reports † PDFs of the Student Score Reports will be available in TOMS. †† LEAs must provide the copy of the CAASPP Student Score Report to each student’s parent/guardian within 20 working days of its delivery to the LEA. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

39. Example 2016–17 Timeline 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

40. Using the Online Reporting System (ORS)

41. Partial Results Available Now in the ORS Use Caution: The results available in this reporting system are partial and may not be a good representation of your school’s or LEA’s final aggregate results. These partial results are not appropriate for public release. Final data will be released publicly by the California Department of Education (CDE) in September 2017. Test results are available three weeks after a student completes both parts—CAT and PT—of a content area. Test results are added nightly. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

42. ORS Summary • The ORS is a Web-based system that displays score reports for each student who has taken the following California assessments: • Smarter Balanced for ELA and mathematics • California Alternate Assessments (CAAs) for ELA and mathematics • Standards-based Tests in Spanish (STS) for Reading/Language Arts (RLA) • Note: All score report data, except for an individual student’s score reports, can be disaggregated into subgroups for detailed analysis. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

43. Features and Reports in the ORS The ORS Score Reports Quick Start Guide is available at: http://www.caaspp.org/rsc/pdfs/CAASPP.ORS-quick-start-guide.2016.pdf Home Page Dashboard Subject Detail Report Claim-Level Detail Report Assessment Target Report Student Listing Report Student Detail Report Manage Rosters 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

44. Users with Access to the ORS For additional information about user roles available within TOMS, refer to the 2016–17 User Role Guidance document on the CAASPP Portal. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

45. Test Results in the ORS CAASPP Portal: http://www.caaspp.org/ ORS User Guide: http://www.caaspp.org/rsc/pdfs/CAASPP.ORS-guide.2016.pdf 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

46. Test Results in the ORS 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

47. Home Page Dashboard 1 2 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

48. Home Page Dashboard 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

49. Home Page Dashboard:Select Test, Administration, and Enrollment Status Select Test 1 • Select Administration • 2014–15 data are final • 2015–16 data are final • 2016–17 are partial Enrollment Status View 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning

50. Home Page Dashboard:Select Grade and Subject 2 Students are in the process of completing tests—results are not yet available. Test results are available for students who have completed both parts—CAT and PT. 2016–17 CAASPP Post-Test Workshops: Using Assessment Results to Inform Teaching and Learning