Transitioning to New Assessments

1. Transitioning to New Assessments Bilingual Coordinator’s NetworkTransitioning to New AssessmentsSTAR and SBACSeptember 20, 2012Patrick Traynor, Ph.D.Director, Assessment Development & Administration Division

2. California Current Assessments • Standardized Testing and Reporting Program (STAR) • California Standards Tests (CSTs) • Early Assessment Program (EAP) • California Alternate Performance Assessment (CAPA) • California Modified Assessment (CMA) • Standards-based Tests in Spanish (STS) • California High School Exit Exam (CAHSEE) • California English Language Development Test (CELDT) • General Educational Development Test (GED) • California High School Proficiency Exam (CHSPE) • National Assessment of Educational Progress (NAEP) • Physical Fitness Test (PFT)

3. Transitioning to New Assessments California Education Code Section 60604.5 requires the State Superintendent of Public Instruction (SSPI) to: • Develop recommendationsaddressing 16 areas toward reauthorization of the assessment system • Develop a plan for transitioning to new “high quality assessments” • Consult with specific stakeholders: • The State Board of Education • Public Schools Accountability Act (PSAA) Committee • Measurement experts from California’s private and public universities • Experts in assessing students with disabilities and English learners • Teachers, administrators, and governing boards from California’s local educational agencies • Parents • Recommendations and Plan due to the Legislature fall 2012

4. “High-quality assessment” • Means an assessment designed to measure a pupil’s knowledge of, understanding of, and ability to applycritical concepts through the use of a variety of item types and formats, including, but not limited to, items that allow for open-ended responses and items that require the completion of performance-based tasks

5. 16 Areas of Consideration The recommendations shall consider including all of the following elements in the reauthorized assessment system: (1) Aligning assessments to standards (2) Implementing common assessments developed by state collaborative (3) Conform to ESEA reauthorization

6. 16 Areas of Consideration (Cont.) (4) Measurement of achievement at a point in time and over time for groups and subgroups of pupils and for individual pupils. (5) Allow for comparison from one year to the next as a reflection of growth over time (6) Valid, reliable, and fair for all students including English learners (ELs) and students with disabilities

7. (7) Assessments of ELs using primary language assessments (8) Ensure no bias with respect to race, ethnicity, culture, religion, gender, or sexual orientation (9) Incorporate a variety of item types including open-ended and performance-based tasks 16 Areas of Consideration (Cont.)

8. (10) Generating multiple measures of pupil achievement, which, when combined with other measures, can be used to determine the effectiveness of instruction and the extent of learning (11) Assess science and history–social science in all grade levels at or above grade 4 (12) Assess understanding and ability to use technology necessary for success in the 21st century classroom and workplace 16 Areas of Consideration (Cont.)

9. 16 Areas of Consideration (Cont.) (13) Formative and interim assessments that provide timely feedback for purposes of continually adjusting instruction to improve learning (14) Use test administration and scoring technologies that will allow the return of test results to parents and teachers as soon as possible (15) Minimizing testing time (16) Including options for diagnostic assessments for pupils in grade 2

10. Types of Assessment • Diagnostic —for learning • Formative — for learning • Interim— for learning and of learning • Summative — of learning

11. Diagnostic Assessment • Ed Code 60603 defines diagnostic assessment as: “ . . . assessment of the current level of achievement of a pupil that serves both of the following purposes: • The identification of particular academic standards or skills a pupil has or has not yet achieved. • The identification of possible reasons that a pupil has not yet achieved particular academic standards or skills.”

12. Formative Assessment Ed Code 60603 defines formative assessment as: “Assessment tools and processes that are embedded in instruction and are used by teachers and pupils to provide timely feedback for the purposes of adjusting instruction and to improve learning.”

13. Interim Assessments Ed Code 60603 defines interim assessments as: “ . . . an assessment that is given at regular and specified intervalsthroughout the school year, is designed to evaluate a pupil’s knowledge and skill relative to a specific set of academic standards, and produces results that can be aggregated by course, grade level, school, or local education agency in order to inform teachers and administrator at the pupil, classroom, school, and local education agency levels.”

14. Summative Assessment Ed Code 60603 defines “achievement test” as: “ . . . any standardized test that measures the level of performance that a pupil has achieved in the core curriculum areas.”

15. Relationship of Assessments Perie, Marion, Gong, Wurtzel, 2007

16. Sharing Assessment Information • Summative Assessments • Information for student, parent, teacher • Information for school and district • State and Federal Accountability • Interim Assessments • Information for student, parent, teacher • Information for school and district • Formative Processes and Tools • Information for student, parent, teacher

17. Common Assessment Formats • Paper and pencil testing • Most common type of format • Used for all assessments • Computer based testing (CBT) • Uses fixed form but is administered using a computer • Computer adaptive testing (CAT) • Presents harder or easier questions, depending on how the student performs as the test proceeds • Can provide greater accuracy with fewer questions

18. Computer Adaptive Testing • The system will adjust the difficulty of items throughout the assessment 18

19. Computer Adaptive Testing20 Items

20. Six Item Types • Selected Response • Short Constructed Response • Extended Constructed Response • Performance Tasks • Technology-Enabled • Technology-Enhanced

21. The Smarter Balanced Assessment Consortium (SBAC) • Area of Consideration 2 (of 16): Implementing common assessments developed by state collaborative • Assessments for grades 3 - 8 and 11 aligned to the Common Core State Standards (ELA and Math) • Operational across Consortium states in the 2014–15 school year

22. 25 states educating approximately 20 million of the nation’s public K to 12 students 21 governing, 4 advisory states An Ever Changing Landscape 22

23. SBAC Procurements* RFP ID RFP Name Contractor RFP Work Released Start Date RFP-00 Project Management Partner WestEd 06/04/2010 10/01/2010 RFP-01 Communications Services Provider GMMB 02/25/201106/03/2011 RFP-02 Technology Readiness Tool Pearson 05/20/2011 12/07/2011 RFP-03 IT Systems Architecture Measured Progress 05/27/2011 09/15/2011 RFP-04 Item Specifications Measured Progress/ETS 07/27/2011 12/01/2011 RFP-05 Psychometric Services ETS 10/07/2011 02/06/2012 RFP-06 Accessibility and Measured Progress 09/22/2011 01/04/2012 Accommodations Policy Guidelines RFP-07 Item Authoring/Item Pool Pacific Metrics 12/09/2011 04/03/2012 RFP-08 Item/Task Materials Development Measured Progress 10/14/2011 02/06/2012 RFP-09Test and CAT Specifications ETS 10/25/2011 02/06/2012 RFP-11/ Test Engine Development for Pilot 03/07/2012 08/20/2012 and Field Test RFP-18/ and CAT Simulations RFP-20 RFP-12Initial Achievement Level CTB/McGraw-Hill 03/14/2012 07/13/012 Descriptors *Status current as of 8/22/12

24. SBAC Procurements (cont.) RFP ID RFP Name Contractor RFP Work Released Start Date RFP-13 Translations 06/15/2012 07/15/2012 RFP-14 Item/Task Writing/Review — Pilot CTB/McGraw-Hill 12/13/2011 03/28/2012 RFP-15 Report Development 04/18/2012 Sept. 2012 RFP-16 Item/Task Development — Field Test Sept. 2012 Jan. 2013 RFP-17 Scoring Nov. 2012 Feb. 2013 RFP-19 Test Administration 04/13/2012 Sept. 2012 RFP-21 Standard Setting Jan. 2014 Apr. 2014 RFP-22 Sustainability Task Force 04/12/2012 Sept. 2012 RFP-23 Formative Assessment Practices Aug. 2012 Nov. 2012 and Professional Learning

25. Logical Components

26. English Language Arts and Mathematics, Grades 3–8 and High School BEGINNING OF YEAR END OF YEAR INTERIM ASSESSMENT INTERIM ASSESSMENT Computer Adaptive Assessment and Performance Tasks • PERFORMANCE • TASKS • Reading • Writing • Math END OF YEAR ADAPTIVE ASSESSMENT Last 12 weeks of year* DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks; model curriculum units; educator training; professional development tools and resources; scorer training modules; and teacher collaboration tools. Computer Adaptive Assessment and Performance Tasks Scope, sequence, number, and timing of interim assessments locally determined Re-take option Optional Interim assessment system— Summative assessment for accountability * Time windows may be adjusted based on results from the research agenda and final implementation decisions. 26 Source: http://www.ets.org

27. School-level Technology Issues 27 Technology-enhanced test items will require compatibility with latest browser technologies Test security will require device to run in “kiosk mode” Devices must support text-to-speech software Test items with multimedia content will require sufficient internal and Internet network bandwidth

28. Technology Readiness Tool Assesses current capacity and comparesto the technology that is expected to be required for new SBAC assessments Areas evaluated: devices, devices to tester ratio, network infrastructure, and staff and personnel Performs inventory, gap analysis and recommended areas where LEAs will need to upgrade technology

29. Technology Readiness Tool Data Collection First data collection window was from March 20, 2012 to June 30, 2012 Subsequent data collections will occur each spring and fall through 2014 Input information for computers expected to be available for online testing in 2014.

30. Technology Readiness Tool Stats (Cont.) 30 • 600,000 devices entered into system • Operating systems inventory • Windows XP: 52% • Windows 7: 20% • Mac OS X: 20% • iOS 5.x: 3% • Others (Vista, Linux, Windows – Other, iOS 4.x, Google Chrome, Android)

31. Technology Readiness Tool Stats (cont.) 31 • Hardware types • Desktops: 67% • Laptops: 19% • Netbooks: 6% • Tablets: 4% • Thin Clients: 4%

32. Guidelines for Purchasing New Hardware • Inform current and future purchases • Covering the vast majority of commercially available computers and tablets • Full guidelines: http://www.smarterbalanced.org/smarter-balanced-assessments/technology

33. California Work with SBAC: Opportunities for Teacher Involvement • Feedback on draft content and item specifications • Writing and reviewing of test items and tasks (2012-13 pilot test; 2013-14 field test) • Range-finding and score validation • Collaborate on designing score reports and Web tools for digital libraries (2013-14) • Scoring of performance tasks (2014-15 and beyond)

34. Current CDE Participation in SBAC Work Groups • SBAC Work Groups • Assessment Design: Item Development • Assessment Design: Performance Tasks • Assessment Design: Test Design • Accessibility and Accommodations • Technology Approach • Formative Assessment Practices and Professional Learning • Deputy Superintendent Deb Sigman executive committee member

35. SBAC Resources • California Department of Education/SBAC presentations be found on the CDE SBAC Web page at http://www.cde.ca.gov/ta/tg/sa/smarterbalanced.asp • Join the SBAC CDE electronic mailing list by sending a blank e-mail to: subscribe-sbac@mlist.cde.ca.gov

36. Assessments to Consider (by content area and grade level)

37. Transitioning to New Assessments California Education Code Section 60604.5 requires the State Superintendent of Public Instruction (SSPI) to: • Develop recommendationsaddressing 16 areas toward reauthorization of the assessment system • Develop a plan for transitioning to new “high quality assessments” • Consult with specific stakeholders: • The State Board of Education • Public Schools Accountability Act (PSAA) Committee • Measurement experts from California’s private and public universities • Experts in assessing students with disabilities and English learners • Teachers, administrators, and governing boards from California’s local educational agencies • Parents • Recommendations and Plan due to the Legislature fall 2012

38. Statewide Assessment Reauthorization Timeline March 21–22: Work Group Meeting May 22–23: Work Group Meeting July 25–26: Work Group Meeting Report writing and review June 12–14: Work Group Meeting September 6: Work Group Meeting State Board of Education Meeting April 17–18: Work Group Meeting Online Survey closes August 31 Launch of Online Survey Regional Public Meetings: 5 meetings between March and April Recommendations to the Legislature in fall 2012 Focus Groups: July - September E-mail Account opened

39. Common Developing Themes • Diagnostic assessments: ELA and Math in grade 2 and grades 3-12 as needed. • Formative Assessment Tools: • SBAC: ELA and Math • Explore the state providing formative tools and processes for science and History/Social-Science • Interim assessments: • SBAC: ELA and Math Grades 3-8, 11 • ELA and Math for all students in grade 2, 9-10 • Science/HSS – state provided computerized interim assessments similar to summative • Summative: • SBAC assessments in grades 3-8 and 11 (ELA and Math). • Summative assessments for science and HSS in grades 3-12, for ELA and math in grade 12, and writing in grades 3-12

40. Common Themes (Cont.) • Matrix sampling in some grades/subjects, specifically science and history–social science (HSS) • Reduce linguistic complexity of assessments to more accurately measure what students know and can do • Items that evaluate critical thinking, problem solving, communication, collaboration, creativity, and innovation • Other measures: student engagement survey, parent survey, graduation rates, attendance • Multiple measures: within classroom, (e.g., tests, quizzes, projects, essays, journaling, class work, portfolios) • Using results to satisfy the high school graduation requirement

41. SBAC Timeline Formative Processes, Tools, and Practices Development Begins Field testing of summative assessment, training school- and district-level staff in formative tools Technology readiness tool available Teams of teachers evaluate formative assessment practices and curriculum resources Full implementation of assessment system Writing and Review of Pilot Items/Tasks (including Cognitive Labs and Small-Scale Trials) Formative tools available to teachers Content and Item Specifications Development Writing and Review Items/Tasks for Field Testing (throughout the school year) Pilot Testing of Summative and Interim Items/Tasks Conducted

42. Patrick Traynor, Director Assessment Development and Administration Division E-mail: ptraynor@cde.ca.gov Jessica Barr, Education Programs Consultant Assessment Development and Administration Division Phone: 916-319-0364 E-mail: jbarr@cde.ca.gov Kristen Brown, Education Programs Consultant (SBAC Coordinator) Assessment Development and Administration Division Phone: 916-319-0334 E-mail: kbrown@cde.ca.gov Reauthorization E-Mail Account reauthorization@cde.ca.gov Contact Information