Higher Order Thinking Skills (HOTS)

1. Higher Order Thinking Skills (HOTS) Teaching and Assessing Higher Order Thinking Skills for K-12 Teachers DAY FIVE: October 25, 2012

2. Review: Higher Order Thinking Skills Standards “HIGHER ORDER THINKING SKILLS include critical, logical, reflective, meta-cognitive, and creative processes, which are activated when individuals encounter unfamiliar problems, questions or dilemmas. Successfully applied, they result in valid explanations, decisions, performances, and products. They facilitate student persistence, self-monitoring, and open-ended, flexible attitudes. Higher Order Thinking occurs when a person takes new information and information stored in memory and interrelates and/or rearranges and extends this information to achieve a purpose or find possible answers in perplexing situations.” • Students will be able to formulate a problem or ask a question. • Students will be able to gather, analyze and check validity of their sources. • Students will be able to actively synthesize information. • Students will be able to construct a well-reasoned argument, or take a well-reasoned position. • Students will be able to connect new knowledge to new or unpredictable situations.

3. Review: Higher Order Thinking Skills Instruction • The 5-E Model (Engage, Explore, Explain, Elaborate, Evaluate) of Inquiry applied to Critical Thinking – “1 billion heartbeats” • Claim/Evidence/Reasoning (CER) applied to Problem-Solving – “Disease Outbreak” • Additional Strategies (clear instructions, examples, modeling, feedback, transitions, questioning) applied to Argumentation and Analytic Reasoning – “Hydraulic Fracturing”

4. Today: Assessing Higher Order Thinking Skills General Principles Formats Assessing Specific Thinking Skills - Examples Using Rubrics Collecting and using samples of Student Work

5. Assessing Higher Order Thinking Skills

6. Assessing Higher Order Thinking Skills: General Principles Specify clearly and exactly what it is you want to assess (align to the HOTS standard, use the language of the standard). Design tasks or test items that require students to demonstrate those skills. Decide in advance what you will take as evidence of the degree to which students have shown this skill, i.e. design rubric. Present something for students to think about – problem, text, scenario, visuals, etc. Use novel material – new to the student. (Difficult to explain to parents!) Distinguish between level of difficulty and level of thinking, and control each separately. Provide students with frequent and descriptive feedback.

7. Assessing Higher Order Thinking Skills Methods / Formats • Selection (multiple choice, matching) • Generation (short answer, essay, performance) • Explanation (giving reasons for selection or generation) Selection Generation Efficiency Breadth Authenticity Depth

8. Assessing Higher Order Thinking Skills “Selection (multiple choice, fixed response) can only be used to assess the rote memory of isolated facts.” Selection: For knowledge, comprehension, application and analysis skills, multiple choice and constructed response formats were generally comparable (Hancock, 1994) Multiple choice items can be used to asses critical thinking skills like “identifying the most plausible assumption, recognizing an author’s purpose, and selecting the most defensible inference,…” (Paul and Nosich, 1992) Bloom (1956) cast both item and answer justification in multiple choice format. “Well-written multiple-choice items can assess higher-order thinking, [but] you wouldn’t rely on multiple-choice items alone for such assessments.” (Brookhart, ASCD, 2010)

9. Assessing Higher Order Thinking Skills Generation: Performance tasks, including laboratory problems, essays, and short-answer constructed-response measures, have been widely recommended as replacements for multiple-choice tests to measure higher order thinking skills. (Wiggins, 1989, 1993) They are domain-specific, can deal with complex, real-life problems and can generate student interest and motivation. They can be costly, time-consuming, and can lack generalizability. Often, assessment items are piloted in “Generation” format, and then the most common generated responses are used to convert the item to “Selection” format.

10. Sample “Selection (with Explanation)” Assessment Item (Critical Thinking)(From ACT Practice Test www.actstudent.org) 26. A student claimed that “If the masses of 1 cm3 of any solid and 1 cm3 of any liquid are compared, the mass of the solid will be greater.” Do the data in Table 1 support his claim? F. No; lead has a higher density than any of the liquids listed. G. No; mercury has a higher density than any of the solids listed. H. Yes; lead has a higher density than any of the liquids listed. J. Yes; mercury has a higher density than any of the solids listed. Table 1 Density Substance Phase (g/cm3) Arsenic solid 5.73 Glucose solid 1.56 Iron solid 7.86 Lead solid 11.34 Zinc solid 7.14 Ethanol liquid 0.79 Ethyl ether liquid 0.71 Glycerol liquid 1.26 Mercury liquid 13.59 Freon-12 gas 0.00495 Krypton gas 0.00343 Methane gas 0.00065

11. Sample “Generation” Assessment ItemNAEP: Using Science Principles NAEP L8.6: Organisms may interact with one another in several ways: They may be in a producer/consumer, predator/prey, or parasite/host relationship… MI GLCE L.EC.06.23 Predict and describe how changes in one population might affect other populations based upon their relationships in the food web.

12. NAEP: Using Science Principles Note: A correct response must include a feasible explanation directly relating the predicted change in robin population to the corn crop failure. Responses do not have to use the terms increase, decrease or same, as long as the explanation is clear.

13. Assessment FormatsStandards best assessed through “Generation” TBAISD Higher Order Thinking Skills Standards: • Students will be able to formulate a problem or ask a question. • Students will be able to gather, analyze and check validity of their sources. • Students will be able to actively synthesize information. • Students will be able to construct a well-reasoned argument, or take a well-reasoned position. • Students will be able to connect new knowledge to new or unpredictable situations.

14. Assessing Students’ Higher Order Thinking Skills • Assessing Students’ Ability to Question (TBA HOTS Standard #1 “… ask a question.”) • Assessing Students’ Ability to Analyze (TBA HOTS Standard #2 “… analyze … sources.”) • Assessing Students’ Ability to Evaluate (TBA HOTS Standard #2 “… check validity of their sources.”) • Assessing Students’ Use of Logic and Reasoning (TBA HOTS Standard #4 “…well-reasoned argument, or … position”) • Assessing Students’ Problem-Solving Skills (TBA HOTS Standard #1 “ … formulate a problem…”)

15. Assessing Students’ Ability to Question Asking questions is essential to developing higher order thinking skills. The ability to ask well-defined questions is an important component of literacy, helping to make students critical consumers of knowledge. Questions can be driven by curiosity about the world, inspired by a model’s predictions, or they can result from a need to provide better solutions to a problem. Some examples: • What exists? – or – What happens? • Why/How does it happen? • How are events related? • What causes what? • How does one know? • What can be done to address a particular need or solve a particular problem? • What tools and technologies are available? How can they help?

16. Assessing Students’ Ability to Question

17. Assessing Students’ Ability to Question Higher Order Thinking • How does the information shown in this graph relate to our changing perceptions of “poverty” over time? • Does the price of a particular technology tend to change as it becomes more widespread? • Why does the slope of each line generally become more steep over time? What does this indicate about our changing society? • The graph stops at 2005. What does it look like now? What will it look like in 50 years? • Why does the graph show declines in “electricity, autos, telephones,” etc. during the 1930’s? • Which inventions tend to parallel each other in their spread throughout the U.S. population? • In 1945, about what percent of U.S. households had a telephone? Lower Order Thinking

18. Assessing Students’ Ability to Question

19. Assessing Students’ Ability to Analyze Analyze: “break something into parts, and see the relationships among the parts.” (Brookhart, 2010) “We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness. That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed, that whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to institute new Government, laying its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness.” - United States Declaration of Independence

20. Assessing Students’ Ability to Analyze Determining the “Main Point, Thesis, or Argument:” Selection: Sample NAEP question, 8th Grade Social Studies 4. According to the passage, the most important purpose of government is to protect a. people from harm. b. the church. c. the truth. *d. people’s rights. (comprehension) 5. Which statement best summarizes the main point being made in the passage? *a. The people should be in control of their own government. b. The church should help governments determine what is right. c. The main function of government is to keep people happy. d. Governments need to be changed regularly to keep them from being unjust. (analysis)

21. Assessing Students’ Ability to Analyze Generation: (with 6-point scoring rubric) What is the main point of the passage above? State the main point in your own words, and then give evidence from the passage.

22. Assessing Students’ Ability to Analyze Compare and Contrast: Sample Assessment: A 4th grade teacher asked her students to choose two objects, identify at least four attributes of each that were alike and four that were different, use a Venn diagram, and then write an essay. The essay was to have an introductory paragraph, a paragraph about similarities, a paragraph about differences, and a final paragraph. Students were to use appropriate transition words and follow the conventions of good writing. The teacher was assessing both higher-order thinking skills and written composition skills.

23. Assessing Students’ Ability to Analyze Student #1: I’m comparing clarinets and saxaphones because I like these instruments. Clarinets and saxaphones are alike because they both are instruments, have reeds, are hard to play. They have to be cleaned regularly. Clarinets and saxaphones are different because, clarinets are black, but saxaphones are gold. Saxaphones are big but, clarinets are smaller. Clarinets need just some air but, saxaphones need a lot. Saxaphones’ reeds are bigger than clarinet reeds. Clarinets are low. Saxaphones are high. I compared clarinets and saxaphones because I like these instruments and the sounds are cool because one is low and one is high. Student #2: My sister and I are different like Morgan has brown hair and I have blond hair. Morgan has brown eyes and I have blue eyes, Morgan has straight hair and I have curly hair. We have different ages too Morgan is twelve years old. And I am ten years old. We are alike in ways too like we are smart, we are tall, we are both funny and we like the same stores. I chose my sister and I because we are alike and different in many different ways.

24. Assessing Students’ Ability to Analyze Student #1: Correctly followed directions, i.e. four paragraphs sequenced correctly, misspelled saxophone, and was incorrect about the pitches of the two instruments. Good with respect to grammar and usage. Clearly, a more academically sophisticated choice for comparison. Student #2: Only three paragraphs. No misspellings, but used less difficult words. Needs to work on her writing, but does understand how to use comparison and contrast. Question: What targeted and helpful feedback would you give to these students? The Point: Be sure to analyze thinking and writing separately!

25. Assessing Students’ Ability to Evaluate Evaluate: to appraise, or critique the material against some criteria (e.g. literary, historical, scientific), which are standard, supplied, or student-generated. It is NOT simply a personal preference, but a well-reasoned conclusion supported with evidence and logic.

26. Assessing Students’ Ability to Evaluate(e.g. Critiquing a Science Experiment) In Maya’s science class, the teacher wanted students to design experiments to find out about temperature changes. Each student was to plan and conduct a simple experiment, measure and graph temperature changes, and write a report about their findings. Maya decided she wanted to study how long it takes coffee to cool. Her teacher said she would bring a coffeemaker to class. Here are Maya’s plans for the experiment. Maya’s Plan: I’m going to ask my teacher to make the coffee. I’ll put it in a cup and I’ll measure its temperature with a thermometer, every five minutes for an hour. I’ll make a line graph of the temperature. I should be able see how fast the line drops as the coffee gets cooler. Do you think Maya’s plan is a good one? Why or why not? Would you suggest any changes to her plan? If so, tell what you would change and explain why.

27. Assessing Students’ Ability to Evaluate Task-specific scoring rubric: (4 points) (Not shared with the students beforehand)

28. Assessing Students’ Ability to Evaluate How could we change the wording of this assessment to make it much clearer to the students? “…design experiments to find out about temperature changes.” “Do you think Maya’s plan is a good one?” Remember: “Specify clearly and exactly what it is you want to assess .”

29. Assessing Students’ Ability to Evaluate(Some Idea Generators in Science) “In your opinion, what is… • the most essential medical discovery?” • the most useful plant / animal on Earth?” • the natural disaster you’d least like to experience?” • most necessary for people to know – biology, chemistry, Earth science, engineering, or physics?” • the most important element in the periodic table?” • the technological invention that has had the greatest impact?” • the most influential living scientist? Dead scientist?” • the best use of mathematics to describe a scientific process?” Explain your evidence. Perhaps provide students with a menu of options, or leave open-ended. Get students involved in making a “rubric” they will use to collect evidence and make their evaluation.

30. Assessing Students’ Use of Logic & Reasoning (A Review) Sound Reasoning: involves determining whether or not an argument is strong, whether its premises or claims are true; and whether they are relevant to and actually support the argument or conclusion; identifying any hidden premises (assumptions), or logical fallacies; and determining whether two or more statements are consistent.

31. Assessing Students’ Use of Logic & Reasoning (A Review) Deductive Reasoning: from a general principle to a specific instance of the principle. True premises guarantee a true conclusion. Inductive Reasoning: from specific instances to a general principle. True premises make the conclusion more likely to be true. Recognizing Logical Fallacies: e.g. Slippery Slope, Black or White, Ad hominem / Genetic, Bandwagon, Appeal to Nature, Appeal to Emotion, Cherry-picking Data, Middle Ground, etc…

32. Assessing Students’ Use of Logic & Reasoning Example: Selection

33. Assessing Students’ Use of Logic & Reasoning Example: Selection Supplied Material: Bill of Rights of the United States Constitution Ms. Gutierrez owns a small house near an interstate highway. She and her family have lived there for three years. The state highway department wanted to put in a new intersection, so they condemned all the property required for construction of the intersection and told the owners that they would pay them each half of what their property was worth. Ms. Gutierrez could challenge the state by citing which amendment? a. Fourth Amendment b. Fifth Amendment* c. Sixth Amendment d. Seventh Amendment

34. Assessing Students’ Use of Logic & Reasoning Example: Generation Supplied Material: Bill of Rights of the United States Constitution Mr. Smith heads a local environmental committee. Mr. Jones wrote a letter to the editor of the local paper and called Mr. Smith’s position “disastrous” for the local economy. The letter also attacked Mr. Smith personally, calling him “ridiculous” and “stupid.” Neither Mr. Smith nor the committee responded, so Mr. Jones threw a rock through the front window of Mr. Smith’s house. Around the rock he had tied the editorial page from the paper, with his letter to the editor printed on it. According to the Bill of Rights of the U.S. Constitution, was Mr. Jones within his rights for any of the actions he took? Did he exceed the boundaries of protected behavior at any point? Explain your reasoning. In your explanation, refer to specific amendments(s) and right(s) and tell how they relate to Mr. Jones’s story.

35. Assessing Students’ Use of Logic & Reasoning Example: Selection “Interpretation of Results” The graph shows that • wealthy people tend to have different political views than do people with less money. • The incomes of certain groups of voters have increased dramatically. • The higher someone’s income is, the more likely he or she is to vote. • Young people are more likely to vote than are older people.

36. Assessing Students’ Use of Logic & Reasoning Example: Generation “Interpretation of Results” What are some plausible explanations for the pattern of voter turnout shown in the graph? Explain why your explanations are reasonable. (Some sample responses) • Higher-income people might have more education and be more likely to recognize the importance of voting • Higher-income people may be more confident in the political system or feel they have more at stake in the system under which they have been successful. • Lower-income people don’t have enough time to vote. • Higher-income people tend to vote more often. • Younger people tend to have lower incomes than older people, and voter participation rates for younger people are less than for older people.

37. Assessing Students’ Use of Logic & Reasoning Example: Generation Algebraic Proofs: Show [(x + 2)3] + 6 = 3(x + 4) = [(x  3) + (2  3)] + 6 Distributive Principle = [3x + (2  3] + 6 Commutative Principle = (3x + 6) + 6 Computation = 3x + (6 + 6) Associative Principle = 3x + 12 Computation = 3x + (3  4) Substitution = 3 (x + 4) Distributive Principle

38. Assessing Students’ Use of Logic & Reasoning Reasoning by Analogy: Historical Parallels that require Inductive Reasoning • Thirteen Colonies declaring independence from Great Britain vs. Confederate States’ secession from the Union; • U.S. involvement in the Korean War vs. involvement in the Vietnam War; • European settlers’ treatment of Native Americans vs. European colonists’ treatment of African or Australian Aborigines; etc…

39. Assessing Students’ Problem-Solving Skills Problem: A goal whose solution is not automatic, and requires thinking! Some “problems” (e.g. balancing chemical equations) are not actually problems, they require procedural knowledge & skill, but little higher order thinking. A good problem solver identifies exactly what the problem is, what might be obstacles to solving it, and what solutions might be expected to work. A good problem solver then tries at least one of the solutions. For more complex problems, a good problem solver can prioritize and evaluate the relative effectiveness of different solutions strategies (Marzano, 1993)

40. Assessing Students’ Problem-Solving Skills The IDEAL Problem Solver: I Identifies the problem D Defines and represents the problem E Explores possible strategies A Act on the strategies L Look back and evaluate the effects of your activities. Bransford and Stein (1984)

41. Assessing Students’ Problem-Solving Skills: Identifying the Problem In a game, Carla and Maria are making subtraction problems using tiles numbered 1 to 5. The player whose subtraction problem gives the largest answer wins the game. Look at where each girl placed two of her tiles. Who will win the game?____________________ Explain how you know this person will win. (released 8thgr math NAEP item)

42. Assessing Students’ Problem-Solving Skills: Identifying the Problem Criteria for feedback or scoring rubric: • Identifies that Maria will win; • Clear, logical explanation based on student’s understanding of place value.

43. Problem Solving – Multiple Strategies A school yard contains only bicycles and wagons like those in the figure above. On Tuesday the total number of wheels in the school yard was 24. There are several ways this could happen. a. How many bicycles and how many wagons could there be for this to happen? Number of bicycles ________ Number of wagons ________ b. Find another way that this could happen. Number of bicycles ________ Number of wagons ________

44. Problem Solving – With Data Released 4th grade Science NAEP item, 2005

45. Problem Solving – With Historical Data New York Times The Gettysburg Address: November 19, 1863 The most well-known speech in American history. But, what did Abraham Lincoln actually say?

46. Problem Solving – With Historical Data Bliss Version: Only manuscript with Lincoln’s signature Bancroft Manuscript: February 23, 1864. Donated to Cornell. Nicolay Copy: Given to Lincoln’s secretary John Nicolay. First page on White House stationery. Hay Draft: Given to Lincoln’s secretary John Hay, donated to Library of Congress, 1916

47. Using Scoring Rubrics • Generic Scoring Rubrics vs. • Item-Specific Scoring Rubrics

48. Generic Rubric – “Writing a Theme” Score of 5: • Thoroughly develops all aspects of the task evenly and in depth • Richly supports the theme with many relevant facts, examples, and details • Demonstrates a logical and clear plan of organization Score of 4: • Develops all aspects of the task but may do so somewhat unevenly • Supports the theme with relevant facts, examples, and details • Demonstrates a logical, clear plan of organization; includes an introduction and a conclusion beyond a restatement of the theme Score of 3: • Develops all aspects of the task with little depth or develops most aspects of the task in some depth • Includes some relevant facts, examples, and details; may include some minor inaccuracies • Demonstrates a satisfactory plan of organization; includes an introduction and a conclusion that may be a restatement of the theme Score of 2: • Minimally develops all aspects of the task or develops some aspects of the task in some depth • Includes few relevant facts, examples, and details; may include some inaccuracies • Demonstrates a general plan of organization; may lack focus; may contain digressions; may not clearly identify which aspect of the task is being addressed; may lack an introduction and/or a conclusion Score of 1: • Minimally develops some aspects of the task • Includes few relevant facts, examples, or details; may include inaccuracies • May demonstrate a weakness in organization; may lack focus; may contain digressions; may not clearly identify which aspect of the task is being addressed; may lack an introduction and/or a conclusion Score of 0: • Fails to develop the task; OR includes no relevant facts or details; OR includes only the theme as copied from the test booklet; OR is illegible; OR is a blank paper

49. Generic Rubric – “Writing a Theme”Subjective or Ambiguous Terms Score of 5: • Thoroughly develops all aspects of the task evenly and in depth • Richly supports the theme with many relevant facts, examples, and details • Demonstrates a logical and clear plan of organization Score of 4: • Develops all aspects of the task but may do so somewhat unevenly • Supports the theme with relevant facts, examples, and details • Demonstrates a logical, clear plan of organization; includes an introduction and conclusion beyond a restatement of the theme Score of 3: • Develops all aspects of the task with little depth or develops most aspects of the task in some depth • Includes some relevant facts, examples, and details; may include some minor inaccuracies • Demonstrates a satisfactory plan of organization; includes introduction and conclusion that may be a restatement of the theme Score of 2: • Minimally develops all aspects of the task or develops some aspects of the task in some depth • Includes few relevant facts, examples, and details; may include some inaccuracies • Demonstrates a general plan of organization; may lack focus; may contain digressions; may not clearly identify which aspect of the task is being addressed; may lack an introduction and/or a conclusion Score of 1: • Minimally develops some aspects of the task • Includes few relevant facts, examples, or details; may include inaccuracies • May demonstrate a weakness in organization; may lack focus; may contain digressions; may not clearly identify which aspect of the task is being addressed; may lack an introduction and/or a conclusion Score of 0: • Fails to develop the task; OR includes no relevant facts or details; OR includes only the theme as copied from the test booklet; OR is illegible; OR is a blank paper

50. Generic RubricCollege and Work Readiness Assessment 6-point scale for each of four categories: • Analytic Reasoning & Evaluation • Writing Effectiveness • Writing Mechanics • Problem Solving