MCAS Standards Validation: High School Introductory Physics

1. MCAS Standards Validation:High School Introductory Physics Sheraton Hotel Braintree, MA September 17-18, 2007

2. Overview of August Standards Setting • Independent standard-setting for: Biology Chemistry Introductory Physics Technology/Engineering • Cut scores recommended for: Failing/Needs Improvement Needs Improvement/Proficient Proficient/Advanced • Each panel composed of: • high school science teachers and administrators • university representatives • community representatives

3. Post Standard-Setting Analysis • Analyzed each of the recommended cut scores (n=12) across all four content areas • Validated 10 of 12 recommended cut scores • Issues related to 2 of the recommended cuts: • Raw score ranges associated with two cuts for F/NI and NI/P for Introductory Physics were extreme outliers • Some Introductory Physics panelists expressed concern about standard-setting process

4. Plenary Session • Welcome • Overview of August Standard Setting Meeting • Post Standard Setting Analysis • Standard Setting versus Standards Validation • STE tests: efforts to establish “equivalence” and uses of test results • Recap of Body of Work (BOW) method and role of performance level descriptors • Questions and Answers

5. Additional Analyses Conducted • Relationship of projected cut scores to psychometric properties of test (TCCs and TIFs) • Relationship of previous student performance in science with that of IP test based on recommended cut scores • Relationship of student work classified as Needs Improvement and Proficient to: • Introductory Physics performance level descriptors • student work classified as Needs Improvement and Proficient for Biology, Chemistry, and T/E

6. Conclusions Post Standard-Setting • Interpretation of Needs Improvement and Proficient diverged significantly from interpretation made in the other content areas • Considered statistical adjustment as remedy but determined Standards Validation preferable given high stakes use of results • Delay in posting of scaled score/raw score conversion tables for all four content areas until standards validation completed

7. Remember, we knew this was going to be challenging… • Standard setting for multiple tests–-where roughly “comparable” standards across all tests is the goal—is technically challenging • Other approaches to standard setting considered: • Overlapping content panels • One panel composed of all representatives for all four content areas • Independent panels

8. General Phases of Standard Setting • Data-collection phase • Policy-making/decision-making phase

9. Standard Setting vs.Standards Validation • Standard setting • Process of establishing original cut scores • Panelists are not provided initial cut points or focused cut point range • Standards validation • Process of validating cut scores • Panelists are provided initial cut points or a narrowed cut point range

10. Introductory Physics Standards Validation Cut score established Cut score needed Cut score needed Warning Needs Improvement Proficient Advanced

11. Standards Validation • Bodies of work to be classified represent a narrowed range of student work based upon: • Standard setting panelists’ recommendations (August) • Psychometric and statistical analyses • Analysis of classification of student work for each of the 12 recommended cut scores (conducted by DOE and Measured Progress content experts) • Pinpointing cut score for Failing/Needs Improvement Needs Improvement/Proficient

12. Massachusetts High School Competency Determination (CD)Requirements: Class of 2010 • Meet local graduation requirements • Attain scaled score of 220 or higher on high school MCAS tests in ELA, mathematics, and STE (or equivalent on MCAS Alternate Assessment) To earn diploma, a student must:

13. Efforts to Establish “Equivalence” of High School STE Tests The High School STE tests: • are based on parallel test design and development processes • are based on performance level descriptors that are comparable in scope and rigor • have been designed to have similar psychometric properties

14. External Validation of Efforts to Establish “Equivalence” of STE Tests • Performance level descriptors used in standard setting have been externally validated by Massachusetts teachers • Technical/psychometric properties analyzed by Dr. Ronald Hambleton, University of Massachusetts, Amherst • Test design and plans for standard setting endorsed by the MCAS National Technical Advisory Committee (TAC)

15. What is the Body of Work Procedure? Standard Setting method where panelists: • examine student work (actual responses to test questions) and performance level descriptors • make a judgment regarding the performance level to which the student work most closely corresponds.

16. Body of Work Fundamentals • Examine the student’s responses to multiple-choice questions • Examine the student’s responses to open-response questions • Judge the student’s knowledge and skills demonstrated relative to the PLDs • Panelists do not need to reach consensus on the classifications

17. Materials Used During Standards Validation • Performance Level Descriptors • General • Content specific • Bodies of Student Work • Responses to constructed-response questions • Multiple-choice summary sheet • Rating Forms

18. General MCAS Performance Level Descriptors Needs Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems. Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems. Advanced Students at this level demonstrate a comprehensiveandin-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems.

19. GeneralSTE Performance Level Descriptors

20. What Next? • Take the Introductory Physics test • Discuss the Performance Level Descriptors • Complete the Item Map • Complete training round • Complete an evaluation form • Complete individual ratings • Receive feedback from first round of ratings • Discuss feedback and provide final ratings • Complete final evaluation form

21. Ground Rules • Role of facilitator is to “facilitate” and keep process on track • Process solely focused on recommending performance standards (cut scores) for MCAS • MCAS performance level descriptorsare integral to process but are not up for debate • Panelists’ recommendations are vital; however, final cut scores determined by the MDOE • Each panelist must be in attendance for the duration of the process for his/her judgments to be considered • Each panelist must complete evaluation form at the end of the event • Cell phones off, please!

22. Agenda Monday, September 17 Breakfast 8:00 am – 9:00 am Plenary 9:00 am – 10:30 am Break 10:30 am – 10:45 am Work session 10:45 am –12:00 pm Lunch 12:00 pm – 1:00 pm Work session 1:00 pm – 5:00 pm Tuesday, September 18 Breakfast 8:00 am – 9:00 am Work session 9:00 am – 12:00 pm Lunch 12:00 pm – 12:45 pm Work session 12:45 pm – until completion

23. Questions?