Misconception or Misunderstanding? Assessing Student Confidence of Introductory Statistics Concepts

1. Misconception or Misunderstanding? Assessing Student Confidence of Introductory Statistics Concepts Kirk Allen, Teri Reed-Rhoads, Robert Terry

2. Background • Statistics Concept Inventory (SCI) • Multiple choice instrument, 38 items • Pilot version, Fall 2002 • Continual revision and testing for four years • Pubs: FIE 2003, ASEE 2004 • Part of larger engineering concept inventory family • Motivated by Force Concept Inventory

3. Motivation of current study • Literature review • Background on difficulties • Attitudes • Reasoning skills: Probability • Piaget • Kahneman & Tversky • Reasoning skills: Statistics • Some teaching strategies

4. Motivation, cont’d. • The reviewed studies are generally very specific and in-depth on a certain topic • Or, they are very general as to why students have difficulties (e.g., attitudes) • Nothing which provides a broad comparison identifying conceptual difficulties across statistics • “Misconception” implies students utilize some thought process • What if they simply lack any conceptual reason for their answers?

5. Method • Online version of SCI • Students rate answer confidence on 1 to 4 scale • 308 students completed the online SCI in Fall 2005

6. Results • Positive trend between confidence and percent correct • Good to know. But… • What’s most interesting is against the trend • Define confidence regions based on rank • Percent and confidence differ by 10 (arbitrary) • Statistical methods to determine confidence regions were inconsistent • Poor classification at extremes • Only looking for a guideline, not necessarily rigorous

7. Results

8. Results • By topic area • Probability most over-confident • Literature suggests misconceptions are common (e.g., Piaget, Kahneman & Tversky) • Descriptive most under-confident • Topics likely encountered elsewhere (e.g., correlation, summary statistics)

9. Over-Confidence A coin of unknown origin is flipped twelve times in a row, each time landing with heads up. What is the most likely outcome if the coin is flipped a thirteenth time? • Tails, because even though for each flip heads and tails are equally likely, since there have been twelve heads, tails is slightly more likely • Heads, because this coin has a pattern of landing heads up • Tails, because in any sequence of tosses, there should be about the same number of heads and tails • Heads and tails are equally likely

10. Graphical representation

11. Discussion • 2nd highest confidence, 2nd lowest % correct • Choice D chosen with very high confidence • The notion of a 50-50 “fair” coin is strongly ingrained • Perhaps to the point of making this a “trick” question

12. Under-Confidence Information about different car models is routinely printed in public sources such as Consumer Reports and new car buying guides. Data was obtained from these sources on 1993 models of cars. For each car, engine size in liters was compared to the number of engine revolutions per mile. The correlation between the two was found to be -0.824. Which of the following statements would you most agree with? • A car with a large engine size would be predicted to have a high number of engine revolutions per mile. • A car with a large engine size would be predicted to have a low number of engine revolutions per mile. • Engine size is a poor predictor of engine revolutions per mile. • Engine size is independent of revolutions per mile.

13. Graphical representation

14. Discussion • Relatively high % correct • Rank 10th easiest, 65% • Positive trend across confidence level • Most common incorrect answer (“C”) is least confident • Two other correlation items have similar trends • Students seem to have correct conceptualization of correlation, though perhaps unaware • Topic likely encountered elsewhere, such as freshman Chemistry lab • Right click  Add trendline  Display R2 on chart

15. Over-confidence • A bottling company believes a machine is under-filling 20-ounce bottles. What will be the alternate hypothesis to test this belief? • On average, the bottles are being filled to 20 ounces. • On average, the bottles are not being filled to 20 ounces. • On average, the bottles are being filled with more than 20 ounces. • On average, the bottles are being filled with less than 20 ounces.

16. Graphical representation

17. Discussion • Increasing % correct as confidence increases • Ignore only n = 5 at confidence 1 • Also high discrimination (0.50) • This doesn’t always “work” • Most common distractor has highest confidence (A) • Some instructors / textbooks favor this definition for null hypotheses, i.e., students may have encountered it.

18. Progress • Update the rating scale to explicitly account for guessing • Added another value as well • Scale now 0 to 5 rather than 1 to 4 • 212 students • More homogenous student population

19. Method #1 • Include guessing in average

20. Method #2 • Exclude guessing

21. Comparisons • Guessing inclusion vs. exclusion • Nothing substantial • Central portion perhaps a little tighter with exclusion • Old data vs. new data • Fewer over / under in new • Nothing switched from over to under or vice versa • Probability still most over-confident

22. Classification

23. Topics (new data)

24. Finale • Presence of probability misconceptions concurs with literature • Students know correlation but they don’t know they know it • Old and new data • Varying the rating scale has little effect • More ideas • Incorporate confidence to guide interviews • Effect of rating on decision making • Overall scores are typical • Try with other engineering concept inventories • Contribute to misconception literature in under-studied subjects

25. Philosophical Aside • A wise man once said… • Those who can't do, teach • Aristotle said… • Teaching is the highest form of understanding.

26. Get in touch: allenk@purdue.edu Learn more: https://engineering.purdue.edu/SCI Purdue Pete says: “How confident are you? Take the SCI Challenge!”