Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience

1. Assessing Quantitative Reasoning in Student Writing: A QuIRKy Experience Nathan D. Grawe Carleton College Serc.carleton.edu/quirk With support from the US Department of Education’s Fund for the Improvement of Post-Secondary Education, the National Science Foundation, and the WM Keck Foundation.

2. What is QR?

3. What is QR? The habit of mind to consider the power and limitations of quantitative evidence in the evaluation, construction, and communication of arguments in public, professional, and personal life.

4. What is QR? Four facets of QR: 1) QR requires a basic skill set

5. What is QR? Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context "The test of numeracy, as of any literacy, is whether a person naturally uses appropriate skills in many different contexts" -National Council on Education and the Disciplines (2001)

6. What is QR? Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves argument

7. What is QR? Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves argument “Deploying numbers skillfully is as important to communication as deploying verbs.” -Max Frankel, The New York Times Magazine

8. What is QR? Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves argument “Numbers [are] the principal language of public argument.” -BBC Program More or Less

9. What is QR? Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves argument 4) QR is a habit of mind “[QR] is not a discipline but a way of thinking….” -Lynn Steen Achieving Quantitative Literacy

10. What do the numbers show?

11. What do the numbers show?

12. How representative is that?

13. Compared to what?

14. How was the variable operationalized?

15. Who is in the measurement sample? Registered Voters Eligible Voters 2006 Midterm Elections

16. Is the outcome statistically significant?

17. What’s the effect size?

18. What’s the effect size? Consider two statements: • A) With a p<.001 we can conclude that whites in MN, WI, and IA were more likely to vote for Obama than whites in LA, MS, and AL • B) The fraction of whites who voted for Obama in MN, WI, and IA was at least 35 percentage points higherthan that in LA, MS, and AL (<15% vs. >50%) –forget the p value!

19. What’s the research design (correllational or experimental)? Fact: Those who work with computers earn 15-20% more than others. “Thus, computer training may, at least in the short run, be a profitable investment for public and private job training programs.”

20. What’s the research design (correllational or experimental)? Other interesting returns: Calculator = 12.8% Telephone = 11.4% Pencil/Pen = 11.2% Work while sitting = 10.1%

21. Controlling for what?

22. Controlling for what? • About 50% of the differential has to do with different career choices…. • 25% percent involves greater time women spend on care-taking…. • The other 25 percent is due to bias and prejudice….”

23. What is QR? “…sophisticated reasoning with elementary mathematics more than elementary reasoning with sophisticated mathematics.” -Lynn Steen Achieving Quantitative Literacy

24. Implications for Assessment Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves communication 4) QR is a habit of mind

25. Implications for Assessment “[QR] is largely absent from our current systems of assessment and accountability.” -NCED (2001)

26. Implications for Assessment Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves communication 4) QR is a habit of mind

27. Implications for Assessment Four facets of QR: 1) QR requires a basic skill set 2) QR demands application in context 3) QR involves communication 4) QR is a habit of mind

28. Implications for Assessment Application in context: “Standardized conditions are decontextualized by design" -Grant Wiggins “'Get Real!': Assessing for Quantitative Literacy ”

29. Implications for Assessment Communication: While traditional assessment tools effectively measure comprehension, the ability to read others’ QR exposition does not guarantee the ability to engage in the creation of QR arguments.

30. Implications for Assessment Habit of mind: “As in book literacy, evidence of students’ ability to play the messy game of the [QR] discipline depends on seeing whether they can handle tasks without specific cues, prompts, or simplifying scaffolds from the teacher-coach or test designer.” -Grant Wiggins “'Get Real!': Assessing for Quantitative Literacy”

31. Implications for Assessment "[QR] requires creativity in assessment, since neither course grades nor test scores provide a reliable surrogate." -Lynn Steen Achieving Quantitative Literacy

32. Implications for Assessment "The interdisciplinary and contextual nature of [QR] cries out for a cross cutting approach." -Lynn Steen Achieving Quantitative Literacy “We want to regularly assess student work with numbers and numerical ideas in the field….” -Grant Wiggins “'Get Real!': Assessing for Quantitative Literacy”

33. The QuIRK Rubric QuIRK’s idea: Measure QR in the “natural” context of papers written in courses across the curriculum—papers written for “authentic” purposes.

34. The QuIRK Rubric Sophomore Writing Portfolio 3-5 papers plus reflective essay written in 2 of the 4 college divisions -observation -analysis -interpretation -documented sources -thesis-driven argument

35. The QuIRK Rubric Writing requirement assessment: 30 faculty members, 3 days, 450 portfolios QR assessment: 6-8 faculty members, 3 days, random sample of portfolios, 1 paper each drawn from analysis, interpretation, or observation categories

36. The QuIRK Rubric 1. A little book-keeping

37. The QuIRK Rubric 2. Did the student take the paper in a QR-relevant direction?

38. The QuIRK Rubric Central Use: Use of numbers to address a central question, issue, or theme Peripheral Use: Use of numbers to provide useful detail, enrich descriptions, present background, or establish frames of reference

39. The QuIRK Rubric The importance of the periphery: “Even for works that are not inherently quantitative, one or two numeric facts can help convey the importance or context of your topic.” -Jane Miller The Chicago Guide to Writing About Numbers

40. The QuIRK Rubric 3. If relevant, to what extent did the student actually use QR?

41. The QuIRK Rubric 4. If present, rate the holistic quality of the QR.

42. The QuIRK Rubric 5. Spotlight on several recurring problems:

43. The QuIRK Rubric 6. Does assignment explicitly call for QR?

44. The QuIRK Rubric Inter-Rater Agreement Statistics QR Relevance: 75% QR Extent: 82% QR Quality: 67% Problem Characteristics: 68%

45. Examples from Student Work Example of QR-irrelevant paper: “The Maiden who Needs No Saving”—an analysis of Keat’s treatment of helplessness and power in “La Belle Dame Sans Merci”

46. Examples from Student Work Example of ineffective peripheral QR paper: Title: Untitled Thesis: A synthesis of 3 alternative views on poverty provides a more complete picture than any 1 perspective alone

47. Examples from Student Writing $192 billion $60 billion 1/65th of GNP 1/230th of GNP

48. Examples from Student Work Example of effective peripheral QR paper: Title: “Les Banlieues d’Islam” Thesis:

49. Examples from Student Work Example of ineffective centrally QR paper: Title: Untitled Thesis: Based on data from a time diary and other sources, the American quality of life is better than that in less-developed societies.

50. Examples from Student Work Example of ineffective centrally QR paper: “Day Care and Development” Topic: Literature review of effects of day care participation on child development