10 Things to Ask Your Local Psychometrician about the Keystone Exams

1. 10 Things to Ask Your Local Psychometrician about the Keystone Exams Dr. Stefan Biancaniello OnHand Schools, Inc. Quality Classroom Consortium October 3, 2012

2. First Some Background Information

3. OnHand Schools Works with Common Assessments • Started in Florida—Hard to Measure Subjects • Next came the Keystone Exams and inquiries from Penn Hills and Elizabeth Forward • Others we are/expect to be working with this year : • Albert Gallatin • Carlynton • Highlands • Clairton • Sto-Rox • Keystone Consortium

4. The Timeline of the Keystone Exams • 2012-13 - Algebra I, Literature, Biology • 2015-16 – English Composition • 2016-17 – Civics & Government, Geometry • 2017-18 – U.S. History • 2018-19 – Algebra II • 2019-20 – Chemistry • 2020-21 – World History

5. The purpose of the Albert Gallatin assessments is… • …to develop Common Benchmark Assessments – similar in content/rigor to the Keystones in Algebra I, Biology, & Literature • ….to develop a pool of items to be used for valid and reliable final exams • ….to develop a pool of items for valid and reliable baseline and mid-course exams • The results will be used ……

6. Now For the 10 Questions for Your Psychometrician!

7. #1What the Heck is a Psychometrician?

8. Psychometrician A psychometrician is someone who practices the science of educational and psychological measurement, i.e., testing. Psychometricians measure the validity, reliability, and fairness of an exam program, and are an integral part in the process of creating valid and reliable tests. Psychometricians are often skilled in statistics, and psychology.

9. #2 What is the Difference between Multiple Choice and Constructed Response Items?

10. The Anatomy of the Multiple Choice Item The Stem Distractor CORRECT ANSWER Distractor Distractor Why are we writing items? • to populate a state item bank • to build common assessments • to work with other teachers • to gain experience writing

11. Constructed Response Items • Constructed response items require students to provide a written response. These questions typically ask students to describe, explain, critique, or evaluate the scenario provided in the stimulus • Constructed response items have multiple correct responses. However, there may be specific content that is required in the response to receive full credit. • Constructed response items can assess one or more benchmarks and range from low to high complexity. • Constructed response items are scored using a rubric

12. Algebra Keystone – from other PDE documents 60% 40%

13. Biology Keystone – from other PDE documents 73% 27%

14. Literature Keystone – from other PDE documents 65% 35%

15. #3How Do You Train Teachers for This?

16. We Teach them A New Language! Try This Exercise • Each of you received a stapled piece of paper when you registered this morning. • Please open the paper now. It contains a Multiple Choice Question written in “Franzipanic” —take just two minutes and figure out the correct answer!

17. The Moral of this Test • With poorly written items, you can get them correct without knowing the intended content. • Reliability issue? • Validity issue?

18. #4How Hard are the Keystone Exams?

19. Field Test Results—Half or More Failed! • Of the 47,000 students who took the Biology exam, 64.1% were NOT proficient: Basic (32.7%) Below Basic (31.4%) • Of the 95,000 students tested in Algebra I, 61.0% were NOT proficient: Basic (41.2%) Below Basic (19.8%) • Of the 42,600 tested in Literature, 49.7% were NOT proficient: Basic(31.4%) Below Basic (18.3%)

20. A recently released problem Tim’s scores for the first 5 times he played a video game are listed below: 4526 4599 4672 4745 4818 Tim’s scores follow a pattern. Which expression can be used to determine his score after he played the video game n times? • 73n+4453 • 73 (n+4453) • 4453n +73 • 4526n

21. Keystone Exam Difficulty • The Keystone Exam Test Questions will almost all be at the Level 2 or 3 from Webb’s Depth of Knowledge • Which leads us to the next question…

22. #5What Is Webb’s Depth of Knowledge________________________________

23. Webb’s Depth of Knowledge • Norman Webb developed a process and criteria for systematically analyzing the alignment between standards and assessments. • Webb’s Depth of Knowledge Model can be used to analyze the cognitive complexity required for the student to master a standard or complete an assessment task.

24. Webb’s Depth of Knowledge • Cognitive complexity refers to the cognitive demand associated with a test item. • The Depth of Knowledge level of the item is determined by the complexity of the mental processing that the student must use to answer the item.

25. Webb’s four levels of complexity Level 1 - Recall- Recall of a fact, information, or procedure Level 2 - Basic Application- of Skill/Concept - Use of information, conceptual knowledge, procedures, two or more steps, etc. Level 3 - Strategic Thinking- Requires reasoning, developing a plan or sequence of steps; has some complexity; more than one possible answer; generally takes less than 10 minutes to do. Level 4 - Extended Thinking- Requires an investigation; time to think and process multiple conditions of the problem or task; and more than 10 minutes to do non-routine manipulations.

26. Cognitive Complexity/ Verbs

27. Same Verb—Three Different DOK Levels Describe DOK 1-Describe three characteristics of metamorphic rocks. (Requires simple recall) DOK 2-Describe the difference between metamorphic and igneous rocks. (Requires cognitive processing to determine the differences in the two rock types) DOK 3-Describe a model that you might use to represent the relationships that exist within the rock cycle. (Requires deep understanding of rock cycle and a determination of how best to represent it)

28. #6 Does Depth of Knowledge Mean Difficulty?_______________________________

29. DOK is NOT about difficulty Difficultyis a reference to how many students answer an item correctly • How many of you know the definition of exaggerate? DOK Low = Recall If all or most of you know the definition , this item is an easy one. • How many of you know the definition of illeist? DOK Low = Recall If most of you do not know the definition, this item is a difficult one.

30. Complexity vs. Difficulty • The Depth of Knowledge levels are based on the complexity of the mental processesthe student must use to find the correct answer, not the difficulty of the item!

31. #7 How Does Webb’s DOK Relate to Bloom’s Taxonomy_________________________________

33. #8Do You Have Item Writing Rules?

34. Item Writing Rules • Yes, we offer training in constructing quality Multiple Choice and Constructed Response Items • A one-page summary explains the rules that we used last week while training in Florida • Shula Nedley and I will be available to discuss this in more detail at one of the breakout sessions today

35. #9Can’t I Just Buy Assessment Questions?

36. Yes You Can Buy or Find Assessment Questions But… • Concern over Alignment • Concern over Depth of Knowledge • Penn Hills Will Discuss Why They Are Creating Their Own

37. #10Is it Worth All This Effort?

38. Yes, It’s Worth the Effort • The Bar is being raised • The Common Core changes the game • The Keystone Exams change the game • The world our students will enter is FAR MORE COMPLEX • We need to raise the Level of our game

39. A Tool To Ease the Pain • Assessment Builder A demonstration of this tool will take place during the breakout sessions later today.

40. Test Your Depth of Knowledge You are trapped in a room with 2 doors, one of which leads to freedom and the other to certain death. You do not know which is which. Each door is guarded by a man, one of whom always lies, while the other always tells the truth. You do not know which is which. You can ask but one question of either man to gain your freedom. Which question should you ask? • How do I get out of this room and stay alive? • Which one of the two door is the correct one? • Who is the man who never tells the truth & always lies? • Which door would the other man say leads to death?