Standard Setting: MCAS High School Science & Technology/Engineering (STE) Tests

1. Standard Setting: MCAS High School Science & Technology/Engineering (STE) Tests Sheraton Four Points Hotel Norwood, MA August 14-16, 2007

2. Tuesday, August 14Overview of Plenary Session • Welcome/Introductions • Overview of MCAS Program and the High School STE Tests • Purpose of 2007 Standard Setting • Body of WorkMethod and Procedures • Ground Rules for Standard Setting • Agenda (Tuesday-Thursday)

3. Department of Education • Sal Beatini, MCAS Science Test Developer • Bob Bickerton, Associate Commissioner • Joyce Bowen, Science Specialist • Katie Bowler, MCAS Science Test Development Lead • Per Christiansen, MCAS Science Test Developer • Kevin Dwyer, MCAS History Test Development Lead • Jake Foster, State Science Coordinator • Mark Johnson, Director of MCAS Test Development • Bob Lee, MCAS Chief Analyst • Matt O’Connor, Administrator for Administration, Analysis and Reporting • Kit Viator, Director of Student Assessment

4. Measured Progress • Ann Adjutant, Physics Developer • Liz Burton, Psychometrician • Lee Butler, Administrative Assistant • Lisa Ehrlich, Assistant Vice President • Kevin Haley, Manager of Data Analysis • Renee Jordan, Service Center Representative • Mark Peters, Program Assistant • Miechelle Poulin, Program Assistant • Michael J. Richards, Program Manager • Kevin Sweeney, Assistant Vice President, Research & Analysis • David Tong, Assistant Director, MCAS Program Management • Karen Whisler, Lead Science Developer • Eric Wigode, Director of MCAS Test Development

5. Biology Panelists Erin Bengiovanni Biology Teacher Joseph P. Keefe Technical Carol Bernon Science Department Head Barnstable High School Sheila Blair Science Dept. Head/Biology Teacher Hull High School Ronit Carter Educational Consultant In Harmony with Education Andrew Creamer Biology Teacher Classical High School Amy Deacon Biology Teacher Pentucket Regional High Jennifer Dube Science Instructor Greater Lawrence Technical High Eileen Dufour Biology II Instructor Greater Lowell Technical High Mary Dulko Biology Teacher Sharon High School Richard Fardy Biology Teacher Wilmington High School Martha Hogan Academic Chair, Science Salem High School Elizabeth Hufnagel Science Teacher Brookline High School Julie Jonelis Biology Teacher Sabis International Charter Sarah Juhlin Science Instructor Blackstone Valley Regional Kelly Kallin Biology Teacher Sharon High School Aaron Mathieu High School Biology Teacher Acton-Boxborough Regional Jim Perry Science Chair Mansfield High School Beth Raccula Science Teacher Walpole High School Dianne Rees Director of Science Braintree High School Julie Robertson Upper School Biology Teacher Foxborough Regional Charter Carla Romney Faculty Member Boston University Deborah Sakelakos MCAS Program Administrator Bedford High School Ernestine Struzziero Science Teacher Lynnfield High School

6. Chemistry Panelists Steven Abdow Chemistry Instructor Durfee High School Thomas Bartosek Biology/Chemistry Teacher Canton High School Catherine Botsford Milne Chemistry Teacher Attleboro High School Patricia Brandl Science Teacher Medford High School Barbara Brigante Teacher Northeast Metro. Regional Stacia Brown Science Curriculum Coord. Oakmont Regional HS Lisa Buron Science Teacher Mansfield High School Tony DiLuna Science Teacher Woburn Memorial High School Mary Duane Science Teacher Tantasqua Regional High School Anthony Fimognari Chemistry Teacher East Longmeadow High School Catherine Francis Chemistry Teacher North Reading High School Boriana Georgieva Chemistry Teacher Swampscott High School Joyce Gleason Educational Consultant Esther Hines Chemistry Teacher Billerica Memorial High School Glenn Horner Executive/Scientist DCE Aprilis Shawn Kenner Chemistry Teacher Sharon High School Ronee Krashes Teacher Framingham High School Peter Nassiff Head of Science Department Burlington High School Diane Perito Chemistry Teacher Malden High School Nitzan Resnick Director South Area Solomon Schecter Day School Michael Thompson Chemistry Teacher Amherst Regional High School Linda Weber Chemistry Teacher Natick High School Tara Yohan Biology/Chemistry Teacher Amherst Regional High School

7. Introductory Physics Panelists Anne Marie Brooks Science Teacher Wareham High School Darren Burris Math Instructor Boston Collegiate Charter Norma Chico High School Physics & Chemistry Doherty Memorial Steven Cogger BMW Technician BMW Peabody Marilyn Decker Senior Program Director, Science Boston Public Schools Michael Doherty Teacher Andover High School Brian Dukeman Science Teacher Whitman Hanson Regional Erin Marie Flanagan Teacher Wachusett Regional High School Cynthia Givens Teacher Andover High School Naila Jirmanus Science Program Coordinator Sabis International Charter Christine John Physics Teacher Belmont High School Mon Luke Teacher Bedford High School Kristin Newton Physics Teacher Cambridge Rindge & Latin Julie Shea High School Science Teacher Revere High School Mary Ellen Stephen Physical Science & Chemistry Teacher Plymouth South High School John Sullivan Retired Physics Teacher Quincy Public Schools Sue Vincent Teacher Turners Fall High School Amy Winston Science Department Chair Newton North High School

8. Technology/Engineering Panelists Joseph Clement Department Head Beverly High School Kevin Connolly Technology Teacher Bedford High School Leigh Estabrooks Inventeams Associate MIT Thomas Fischer Technology Eng. Teacher Hopedale Jr. Sr. High School Pamela Glass Math/Science Curriculum Coordinator Westport Community School Ramiro Gonzalez Engineering & Technology Teacher Boston Arts Academy William Hankey Engineering Teacher Doherty Memorial High School Rebecca Lothrop Teacher Clearway School Charles Pouliot Teacher Lawrence Public Schools Lee Pulis Curriculum Developer Museum of Science Bruce Rawley Technology/Engineering Teacher Millbury High School Anthony Ruscito Technology Teacher Bedford Public Schools Lynn Russell Biology Teacher New Leadership Charter Mike Stevens Technology/Engineering Teacher Maynard Public Schools Victor Valenti Retired Engineer Compaq Catherine Wolf High School Science Teacher Brookline High School

9. Standard Setting Facilitators • Biology – Lisa Ehrlich • Chemistry – Margaret Hill • Introductory Physics – Dona Carling • Technology/Engineering – Tim Crockett

10. Purpose of MCAS Program • Inform/improve curriculum and instruction • Evaluate student, school, and district performance according to Curriculum Framework content standards and MCAS performance standards • Determine eligibility for high school diploma (Competency Determination)

11. Massachusetts High School Competency Determination (CD)Requirements • Meet local graduation requirements • Attain passing standard (scaled score of 220) or higher on high school MCAS tests or equivalent on MCAS Alternate Assessment To earn diploma, a student must:

12. Selected Features of MCAS • Custom developed based on Massachusetts Curriculum Framework content standards and MCAS performance standards • 100% of questions used to determine student scores released annually • Measures performance of ALL students educated with public funds • Results reported according to scaled scores and performance levels

13. Grade 3 Reading, grade 6 Math, and grade 7 ELA tests introduced Massachusetts Education Reform Law passed Newly required NCLB tests (grades 3-8) introduced 2001 1993 2006 1998 2003 2010 First MCAS operational tests introduced (ELA, Math, and Science & Technology, grades 4, 8, and 10) Class of 2003 first graduating class required to earn a CD (ELA and Math) STE tests in grades 5 and 8 introduced Class of 2010 first graduating class required to earn a CD in STE Historical Background of the MCAS Tests

14. Development of Massachusetts STE Curriculum Framework (CF) Standards and Assessments 1996 First STE Curriculum Framework introduced1998-2000 Integrated HS STE test administered2001-2002 Revised STE Curriculum Framework (major revisions); no STE testing2003 Question tryout for tests in Biology, Chemistry, Introductory Physics, and Technology/Engineering; no results reported2004-2005 Pilot testing of STE tests; item-level results only BOE decides class of 2010 to pass one of STE tests to earn CD2006 High School STE CF revised (minor revisions)2007 Operational test 9th graders may earn CD by passing one of four STE tests (ELA and Math requirements also apply)

15. MCAS STE Testing Program • End-of-course tests offered in grades 9 and 10: • Biology • Chemistry • Introductory Physics • Technology/Engineering • Students required to pass one of four tests in order to earn CD

16. High School STE MCAS Test Design Each of the four tests includes: 60 total raw score points = MCAS Scaled Scores/Performance Levels 200 220 240 260 280

17. High School Science and Technology/ Engineering Test Reporting Categories

18. Science and Technology/Engineering Test Reporting Categories

19. Rationale: End-of-Course STE Test Menu • Promote deeper assessment of content than integrated test • Support diverse course offerings and scheduling flexibility for high schools • Provide students choice of assessment for CD requirement

20. Efforts to Establish “Equivalence” of STE Tests The STE tests: • are based on parallel test design and development processes • are based on performance level descriptors that are comparable in scope and rigor • have been designed to have similar psychometric properties

21. External Validation of Efforts to Establish “Equivalence” of STE Tests • Performance level descriptors used in standard setting have been externally validated by Massachusetts teachers • Technical/psychometric properties analyzed by Dr. Ronald Hambleton, University of Massachusetts, Amherst • Test design and plans for standard setting endorsed by the MCAS National Technical Advisory Committee (TAC)

22. Content Standards vs. Performance Standards • Content standards = “What” Describe the knowledge and skills students should acquire in a particular content and grade • Performance Standards = “How well” Describe student work on MCAS tests at the Needs Improvement, Proficient, and Advanced levels

23. General MCAS Performance Level Descriptors Needs Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems. Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems. Advanced Students at this level demonstrate a comprehensiveandin-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems.

24. GeneralSTE Performance Level Descriptors

25. Purpose High School Science & Technology/ Engineering: To establish cut scores for MCAS performance levels Cut score needed Cut score needed Cut score needed Failing Needs Improvement Proficient Advanced

26. Linking Performance Standards with Student Work • What is standard setting? Establishment of cut scores to distinguish between performance levels • What is your job? Use the PLDs to evaluate student work and make recommendations for where cut scores should be set

27. Purpose of Standard Setting • Determine cut scores for reporting assessment results • Answer the question: • How much is enough?

28. General Phases of Standard Setting • Data-collection phase • Policy-making/decision-making phase

29. Selected Standard-Setting Methods • Angoff • Bookmark • Body of Work

30. Choosing a Standard-Setting Method • Prior usage/history • Recommendation/requirement by policy-making authority • Type of assessment Body of Work method chosen for MCAS tests in High School Science & Technology/Engineering

31. What is the Body of Work Procedure? Panelists examine student work (actual responses to test questions) and make a judgment regarding the performance level to which the student work most closely corresponds. HS STE Standard Setting: Panelists examine student work that has not been previously classified and determine how that work should be classified.

32. How to Classify Student Work Materials you will need: • Performance Level Descriptors • General • Content specific • Bodies of Student Work • Responses to constructed-response questions • Multiple-choice summary sheet • Rating Forms

33. How to Classify Student Work • Examine the student’s responses to multiple-choice questions • Examine the student’s responses to open-response questions • Judge the student’s knowledge and skills demonstrated relative to the PLDs • Panelists do not need to reach consensus on the classifications

34. How to Classify Student Work To help prepare you to do these ratings, you will spend time becoming familiar with the following: • Test • Performance level descriptors • It is important that all panelists have a clear, common understanding of the PLDs. • Bodies of student work • Multiple-choice items • Constructed-response items

35. How to Classify Student Work • You will have the opportunity to discuss your classifications and change them if desired. • Don’t worry! We have procedures, materials, and staff to assist you in this process.

36. What Next? Break into content-area groups: • Take the assessment • Discuss the Performance Level Descriptors • Complete the Item Map • Complete training round • Complete individual ratings • Receive feedback from first round of ratings • Discuss ratings and revise • Receive feedback from revised ratings • Discuss feedback and provide final ratings • Complete an evaluation form

37. Top 8 Most Misunderstood Things about Standard Setting 5. We should use this time to rework HS Science and Tech/Eng. performance level descriptors. 8. Standard setting is a great opportunity to rewrite Curriculum Framework standards. 7. The process is rigged. 6. This is a good time to vent about all the things you hate about MCAS.

38. Top 8 Most Misunderstood Things about Standard Setting 1. Disagreement is bad. 4. Standard setting is scoring. 3. Only HS Science and Tech/Eng. scholars should be doing this work. 2. Only teachers should be doing this work.

39. Ground Rules • Role of facilitator is to “facilitate” and keep process on track • Process solely focused on recommending performance standards (cut scores) for MCAS • MCAS performance level descriptorsare integral to process but are not up for debate • Panelists’ recommendations are vital; however, final cut scores determined by the MDOE • Each panelist must be in attendance for the duration of the process for his/her judgments to be considered • Each panelist must complete evaluation form at the end of the event • Cell phones off, please!

40. Agenda Tuesday, August 14 Plenary 9:00 am – 10:30 am Break 10:30 am – 10:45 am Work session 10:45 am –12:00 pm Lunch 12:00 pm – 1:00 pm Work session 1:00 pm – 4:00 pm Wednesday, August 15 Breakfast 7:30 am – 8:30 am Work session 8:30 am – 12:00 pm Lunch 12:00 pm – 12:45 pm Work session 12:45 pm – 4:00 pm

41. Agenda Thursday, August 16 Breakfast 7:30 am – 8:30 am Work session 8:30 am – 12:00 pm Lunch 12:00 pm – 12:45 pm Work session 12:45 pm – Until completion

42. Room Assignments Physics: 101 Chemistry: 102 Biology: 103 Tech/Eng: 104

43. Questions?