Using Standard Setting to Set CTE Technical Skill Attainment Measures and Improve Understanding and Trust of CTE Stakeho

1. Using Standard Setting to Set CTE Technical Skill Attainment Measures and Improve Understanding and Trust of CTE Stakeholders Presented by Denise Sibley Laura Jean Kerr Mississippi Assessment Center Research and Curriculum Unit

2. Measuring Technical Skill Attainment • Requirement established by Perkins IV • Assessments have been developed • Standard setting needs to be established

3. What is Standard Setting? The proper following of a prescribed, rational system of rules and procedures that result in determining: • cut scores and performance levels • what’s tested • proper statistical analyses

4. Benefits of Standard Setting • Ensures that decisions are based on high-quality data • Ensures that data are combined in a systematic, reproducible, objective, and defensible manner • Increases understanding and trust on the part of the stakeholders

5. Steps for Setting Performance Standards

6. Choosing a Standard-Setting Method • Researched standard setting methods • Limited literature for CTE assessments – lots of academic literature • Method used had to present a strong validity argument for the results • Method used had to be suitable for the CTE content • Method used had to be straight-forward

7. Item Descriptor Matching • Performance level descriptors (PLDs) need to developed • Descriptions of the performance levels into which examinees will classified (e.g., Minimal, Basic, Proficient, or Advanced) • Participants rate items according to PLDS • Items with agreement less than .67 are re-rated • Use results to distinguish between the four performance level categories.

8. Prepare Descriptions of the Performance Categories • PLDs should be specific to each standard in the curriculum • Challenging because there were over 100 curricula • Developed Career and Technical Education PLDs

9. Career and Technical Education PLDs • Advanced: An advanced student evaluates and integrates transferable academic/workplace knowledge and skills in multiple situations as a productive contributor in the workplace. • Proficient: A proficient student acquires and applies academic/workplace knowledge and skills to become a productive contributor in the workplace. • Basic: A basic student has limited acquisition and comprehension of academic/ workplace knowledge and skills that are necessary to become a productive contributor in the workplace. • Minimal: A minimal student has not developed academic/workplace knowledge and skills that are necessary to become a productive contributor in the workplace.

10. Program Description and Standards from Curricula Included program descriptions and standards from the curricula as a reference for participants to ensure they aligned their ratings with the curriculum

11. Selecting Subject Matter Experts (SMEs) Used stratified random sample method for selecting participants–geographic location based on 15 community colleges Randomly selected teachers from each geographic location by program area Had to be able to attend standard setting meeting Approximately 400 subject matter experts participated

12. Materials for Standard-Setting Meeting • PLDs • Standards from each curriculum framework • Program description from each curriculum framework • Test item booklet with only field-tested items • Item difficulty is used to determine the range for each performance level • Sign-in sheets with logins and passwords • Bubble sheets for backup if site networks went down

13. Conducting the Meetings • Facilitators were trained prior to the meetings to insure that meetings were conducted in a consistent manner • Meetings were held across the state over the course of 3 days • A Web interface was developed for participants to enter their responses • Scheduled participants from each program area to meet on the same day • Participants received instructions on using the materials and the Web interface • Participants went through test item booklets and matched items to PLDs and also were able to write comments about the items • When all participants submitted their responses, they were calculated and any items that didn’t have a .67 agreement were re-rated. • Participants completed a survey after they completed the meeting

14. Using Technology • Conduct Standard Setting over long distances • Coordinate Standard Setting for the best use of Subject Matter Experts time • Keep up with ever-changing needs of Career and Technical Education • Organize and compile data

15. Consistency in Standard Setting • Consistency is the justification for subjective limitations • Training of Subject Matter Experts • Standard Setting environment • Policies and procedures • Consistency contributes to quality data

16. Survey Comments • Communication • “The PowerPoint would have been helpful if I had seen it before today.” • “I did not fully understand the descriptors. I probably over estimated the students' ability.” • “The performance level indicators are open to many interpretations.” • Lesson learned: • Communicate with SMEs early and often.

17. Survey Comments • Meeting Environment • “It was the room and atmosphere where the training was held. It was in a room off the cafeteria. Which means...guess what...too much traffic. We could hear the phone conversations next door from an office, when the phone was ringing and etc.” • “Could use a break for the 200 questions and refreshments.” • “People who arrive late should not be able to enter.” • Lesson Learned: • Every aspect (even the smallest detail) of the testing environment should be taken into consideration.

18. Survey Comments • Technology • “I would like to see the questions online with the performance descriptors. I believe the process would go a lot faster and would allow for easier comments.” • “Website does not allow you to go back and alter responses on previous pages in event of mistakes.” • “Love doing this on computer!” • Lesson learned: • Incorporated these comments into development of new interface

19. Survey Comments • Trust and Understanding • “I appreciate being able to give feedback on the questions. This is the first time I have been invited to do so.” • “I think that this is a step in the right direction for standardized testing.” • “Good program...would like to know more about how the process works with our peers, etc.” • Keeping stakeholders in the “loop” fosters trust

20. Final Steps • Compile data • Provide feedback to stakeholders • Document process • Improve process • Publish results

21. Contact Information Denise Sibley Interim Coordinator of Assessment and Accountability denise.sibley@rcu.msstate.edu Laura Jean Kerr Assessment Specialist laura.kerr@rcu.msstate.edu