Fall 2014 Test Coordinator Training End of Course Exams ( EoCs )

1. Fall 2014 Test Coordinator Training • End of Course Exams (EoCs)

2. Purpose of an EoC Exam • Statewide curriculum alignment. • Determining and measuring teacher and school leader effectiveness—Educator Effectiveness System (EES) (6.60.8 NMAC). • The evaluation system must include measures of student achievement growth (50 percent): • SBA/PARCC—35% • Department approved assessments—15% • Providing an Alternative Demonstration of Competency (ADC) in the academic content standards required for graduation (6.19.7.7 NMAC).

3. Developing an EoC Exam • Step 1: Identify the construct, domains, and indicators to be measured – standards based assessment. • The knowledge, skills and abilities proficient students should know and be able to do at the end of a course or grade.

4. Developing an EoC Exam • Step 2: Develop a Table of Specifications. • Table of Specifications becomes the concrete assessment plan for the EoC exam. • What type of assessment is most appropriate for each domain? • What is the cognitive difficulty or cognitive demand for each domain? • How many items are necessary to comprehensively assess the knowledge, skills and abilities for each domain of the assessment?

5. Developing an EoC Exam • Step 3: Develop the Assessment • Groups of teachers from across the state. • Training on item writing. • Collectively create items. • Produce test form. • Create and verify answer key. • Produce rubrics and administration guide for all constructed response and performance items.

6. Developing an EoC Exam • Step 4: Content/Bias Review • Groups of teachers from across the state. • New and Previous • Training • Review blueprints • Review test items for content and bias • Verify logic of test form • Verify answer key

7. Developing an EoC Exam • Step 5: Give the test and submit data. • Tests will be available online on Blackboard beginning spring 2015 • (Yay!) • Pilot fall 2014 (select districts/schools) • No cost to districts/schools • More information forthcoming

9. Why is Submitting EoC Data Important? • Student Level Data is Used for • Standard Setting (passing scores for new tests) • Item Review • Student Records for passing an exam • Alternative Demonstration of Competency for Graduation • Educator Effectiveness System • Provides larger sample size

10. Data Submission Two Required Submissions: • SOAP Item Data • Only for Select EoCs • Used for setting passing scores • Minimum 50 RANDOM student scores per test • STARS Overall Score • For ALL EoCs • For ALL Students

11. EoC Reports in STARS

12. STARS Reports • EOC Submission Validation Report • Student level validation errors for tests. • Report used by districts to validate 40, 80, 120 and End of Year EOC submissions. • Becomes a formal process for Fall 2014

13. STARS EoC Report Elements • District Name • District Code • Location Name • Location ID • Student ID • Student First Name • Student Last Name • Subject • Subtest Identifiers • Grade From • Grade To • Student Grade Level • Max Score • Raw Score • Max Tests • Subject Count • Date Test Active • Date Test Inactive • Test Date • Exists Error • Date Error • Score Error • Grade Error • Max Test

14. Data Quality is Essential! • Data needs to be accurate • Follow instructions given by PED • Data Submission Guide • STARS Manual • Not sure—Ask! • Errors create delay in results • STARS EoC data submissions tied to 40/80/120/EOY day reporting

15. Developing an EoC Exam • Step 6: Data Analysis • P-values • IRT analysis • Possible elimination of items • Flag items for further review

16. Developing an EoC Exam • Step 7: Standard Setting • Groups of teachers • New and Previous • Set passing scores • Bookmark Method • Body of Work Method

17. Developing an EoC Exam • Step 8: Content Review #2 • Groups of teachers • New and Previous • Review flagged items • Consider district/school/DTC/STC feedback

18. Developing an EoC Exam • Step 9: Re-version and re-administer • Any change  new version • Minor changes, passing score remains • Major changes, new passing scores must be set

19. End of EoCs • New Version—version control • Adoption of new standards—Common Core • The performance does not meet scientific standards • Item level analysis • Reliability • pValues

20. EoC Procedures • Current: paper tests • Upcoming: online hosting on Blackboard • December: Blackboard pilot with select • districts • Spring: Widespread Blackboard • administration • Paper administration will be permitted. • Data will be automatically be submitted to PED!

23. Summer 2014 EoCs

24. Summer 2014 EoCs

25. EoC Administration Windows

26. EoC Test Structure • Most Tests: Multiple Choice • (request from AAAC) • Elective Courses: Performance Components • HS: 90 minutes • MS and ES: 60 minutes

27. EoCs: High Stakes Tests • Treat as Final Exams • Security Procedures are Essential! • Directions for Administration (DFA): administration procedures, teacher access, use of calculators, use of scratch paper, make-ups, accommodations, materials chain of custody, scoring requirements, administration script

28. EoCs: High Stakes Tests • Testing Irregularities: submit to PED • DFA administration procedures will change with implementation of CBT. • Security will be easier to maintain.

29. EoCs: Student Preparation • Teachers: • Access Blueprints (on PED website) • PBA: Access administration guides (on PED website) • DFA (on PED website) • NAEP Questions Tool (on NAEP website)

30. EoC Questions? EoC Data Judy Harmon judith.harmon@state.nm.us EoC Procedures Joslyn Overby joslyn.overby@state.nm.us