Assessing Critical Thinking and Quantitative Reasoning in MA 103

1. Assessing Critical Thinking and Quantitative Reasoning in MA 103 Gerald Kruse and David Drews Juniata College Huntingdon, PA kruse@juniata.edu drews@juniata.edu

2. History • Experimental Design • Data • Tentative Plans • Comments/Suggestions Outline of Status Report/Mentoring Session

3. MA 103, Quantitative Methods, aka “QM”“Mathematics 103 prepares students to be quantitatively literate citizens in today's world. By learning to think critically about quantitative issues, students will be able to make responsible decisions in their daily lives. Problems are analyzed and solved using numerical, graphical, statistical, and algebraic reasoning. Technology is used to help visualize data and facilitate calculations, as well as to present quantitative output and verbal arguments. ” • Collegiate Learning Assessment, (CLA), Performance Task Goals“Critical Thinking, Analytic Reasoning, and Problem Solving“Written Communication”

4. MA 103, Quantitative Methods at Juniata College • Juniata has a Quantitative Skills Requirement • Non-majors course: MA 103, Quantitative Methods • Pre and Post Assessment (Skills and Attitudes) -55 min exam given on the first and last day of semester - Fall 2009 transition from math skills to CLA performance task • Three Projects during the semester - began using CLA performance tasks Spring 2009 - one class period to present and start, then due over one week later - students appreciate open-ended assignment

5. Assessment Schedule, Fall 2009

6. Higher-Order Skills Assessed in Rubric • Evaluating Evidence • Analysis / Synthesis / Conclusion • Presenting / “creating” evidence • Acknowledging alternatives to THEIR conclusion • Completeness • Mechanics/Persuasiveness

7. Evaluating Evidence Category on Rubric Identifying relevant evidence, evaluating evidence credibility, reliability, relevance • Not Attempted (0) • Emerging (1,2)Mentions one or two documents, with:- No or wrong evaluation on both (1)- cursory-to-OK eval on document C, flawed on other (2) • Developing (3,4)Mentions two documents (one must be C), with:- cursory-to-OK evaluation on both (3)- good evaluation on both (4) • Mastering (5,6)Evaluation of C is good, and evaluates two other doc with:- acceptable evaluations (5)- good evaluations (6)

8. Rubric Reliability • DimensionQuestion 1Question 2 • % spot on% +/- 1 % spot on% +/- 1 • Evaluation27.3 81.1 54.6 95.4 • Conclusions50.086.4 45.5 81.8 • Create 68.2 90.9 95.5 100.0 • Alternatives 77.3 100.0 68.2 81.8 • Completeness 63.4 95.4 50.0 81.8 • Mechanics86.4 100.0 77.3 100.0 • Overall: spot on = 63.6% +/- 1 = 91.3 %

9. Rubric Reliability, continued • Question 1 total score: r = .713, p = 0.000 • Question 2 total score: r = .828, p = 0.000 • Total score: r = . 873, p = 0.000

10. Question 1 Mean by group and pre/post

11. Question 2 Mean by group and pre/post

12. Total Score Mean by group and pre/post

13. Possible changes for Spring 2011 • Explore the connection between the critical thinking rubric and assessments/grades used for the in-class tasks • Provide more critical thinking feedback to the students • More attention in class on the module analyzing sources (some low scores on the pre/post assessment were a result of students accepting flawed documents, done the day before fall break in 2009…) • Your ideas here…

15. Analysis/Synthesis/Conclusion Category on Rubric Identifying relevant evidence, evaluating evidence credibility, reliability, relevance • Not Attempted (0) • Emerging (1,2)Incorrectly implies , or states directly, agrees that “banning aspartame would improve the health of the state’s citizens” with:- no evidence (1)- evidence (2) • Developing (3,4) Implies, or directly disagrees, with Sauer, noting inconsistency of claim with data in doc C but reason is:- inaccurate/unclear or incomplete(3)- good (4) • Mastering (5,6) - Says C doesn’t support claim and is clear about reason and uses F reasonably well (5)- Satisfactorily uses conditional probability when discussing relationship between headaches and aspartame usage (6)

16. Performance Task Scenario for Pre and Post-Assessment • Sen. Nathan Dulce is running for re-election vs. Pat Sauer • Proposed bill to ban aspartame, an artificial sweetener, from being added to any soft drink or food product, Dulce opposes, Sauer approves. • Pat Sauer made two arguments during a recent TV interview: • (1) Strong correlation between the number of people who consume aspartame and headaches, so,“banning aspartame would improve the health of the state’s citizens.” (2)“Aspartame should be banned and replaced with sucralose.” Pat Sauer supported this argument by referring to a news release.

17. Questions on Pre/Post Assessment • Question #1 • Pat Sauer claims that “banning aspartame would improve the health of the state’s citizens” (Documents E contains the chart he presented, and it is based on data from the tables in Document C). What are the strengths and/or limitations of Pat Sauer’s position on this matter? Based on the evidence, what conclusion should be drawn about Pat Sauer’s claim? Why? What specific information in the documents led you to this conclusion? • Question #2 • Pat Sauer claims that “aspartame should be banned and replaced with sucralose.” What are the strengths and/or limitations of Pat Sauer’s position on this matter? Based on the evidence, what conclusion should be drawn about Pat Sauer’s claim? Why? Is there a better solution, and if so, what are the its strengths and/or limitations? Be sure to cite the information in the documents as well as any other factors you considered (such as the quality of the research conducted on aspartame) that led you to this conclusion.

18. Critical thinking involves evaluating/making good arguments Making good arguments involves. . . • Clearly stating a conclusion • Evaluating and selecting evidence • Creating links between evidence and conclusion We can then consider quantitative reasoning as critical thinking involving numbers/data…