Rubric-based Assessment of Critical Thinking Skills

1. Rubric-based Assessment of Critical Thinking Skills Guy Hoelzer and Christie Howard Department of Biology UNR or

2. More than “Just the Facts, Ma’am”:Assessment of Critical Thinking for Biology Majors and Programs Guy Hoelzer and Christie Howard Department of Biology UNR

3. Our Motivating Factors • Critical thinking (CT) has been identified by the Biology faculty as one of the key things we want our majors to be able to do well upon graduation • CT is notoriously difficult to teach and assess • Not part of our normal course content • Emphasis on CT in the capstone course for Biology majors leaves the impression that they are not very good at CT and don’t really understand what we mean by it

4. 4. Indeed, they don’t seem to trust the instructor’s explanation of CT • It seems too subjective, so the target is the mind of the instructor rather than anything of value to the student • It seems that every instructor means something different by CT, so it doesn’t seem like a real or particular kind of skill • Instructors too often pay lip-service to CT without giving it weight in grading

5. 5. Generic CT Skills vs. Discipline-Specific Skills? • We would like to generate majors who “think like a biologist” • On the other hand, CT skills ought to be generally useful in any intellectual endeavor

6. 6. Student Assessment Of CT Skills Is Hard Enough - Can We Do Programmatic Assessment Of CT? We would like to engineer a curriculum that builds on student CT skills in an integrated and coherent fashion

7. We Propose To … emulate the rubric-based institutional model developed at Washington State University [see http://wsuctproject.wsu.edu/]

8. The CT Rubric for Biology Students • Will include the key components of CT identified by the Biology faculty • Will serve as a template that can be modified to meet the needs of specific courses • Will be a primary tool for assessing student work at two or more points within the major’s curriculum • Track development of the CT skills of individual students • Focus on an Intro course (Howard) and the capstone course (Hoelzer)

9. Developing the Rubric • H&H will design a 1st draft of the rubric based on the relevant assignments in their respective courses • Students are asked to criticize scientific papers in each course • H&H will do a pilot study by using the1stdraft of the rubric to assess 20-30 sample student papers from each course • Does the rubric effectively discriminate student performance over the full range encountered in both courses?

10. A Faculty Retreat • Introduce the rubric to the Biology faculty (from both UNR and TMCC) and solicit feedback at a faculty retreat • TMCC will be included to facilitate articulation for transfer students • Ideally, the rubric will be useful and ultimately used in TMCC courses as well as UNR courses • Outside evaluators (experts) will be present to contribute to the discussion and report to H&H on the efficacy of our process • Evaluators, like John Mahaffy, will be consulted throughout the duration of the project

11. Consistency in Rubric Scoring • H&H will design a second draft of the rubric based on feedback from the retreat • Clarify and calibrate the rubric with an Inter-rater reliability (IRR) study • Each member of the IRR team will score the same set of 20-30 pieces of student work • Meet to discuss differences among raters and possible causes of those differences • H&H will again revise the rubric based on the IRR results to maximize repeatability and tune the scoring scale to student variances

12. Another Faculty Retreat • Explain the revised rubric to the faculty • Workshop in which all faculty will use the rubric to score student work • Faculty training • Another round of faculty input into rubric development • Effectively another (informal) IRR study • Fostering faculty “buy-in” on use of the rubric

13. A Novel Student IRR Study • It is just as important that the meaning of the rubric be clear to students (consistently interpreted among students) as to the faculty • Student interpretations should be the same as the intended meaning • Students should recognize a consistent thread regarding CT across courses and instructors • A team of students will use the rubric to score anonymous student work in an IRR study

14. Applying The Rubric • Sample work from the Intro and Capstone courses will be collected and assessed • As the project ages, we will collect repeated samples from individual students where possible • We will aim to codify a comparable use of the rubric in one or more 300-level courses from the Biology major’s curriculum and encourage use of our CT rubric wherever appropriate throughout the Biology curriculum

15. Work by the Assessment Team • Team will meet after each semester to assess at least 30 papers from each course • Each meeting will start with a small IRR study to align the scoring practices of team members • Each piece of student work will be scored by two team members, and discrepant scores will be resolved by a third team member • We have proposed to pay all participants in this project for 3 years • We hope that use of the rubric by the faculty for both student and programmatic assessment will become valued and routine, and easily incorporated into ordinary departmental activities when the grant has ended

16. Some Planned Data Analyses • Track patterns of growth in student CT skills • Compare performance in CT skills with other variables • E.g., GPA, number of CT-type Biology courses taken, number of CLA courses taken • Compare 1st generation college student performance with others • Compare transfer with non-transfer students • Compare assessed CT skills with the perceptions of our alumni and their employers

17. Refine Our Curricular Map • Obtain a more detailed understanding of the extent to which CT skills are taught and/or assessed in particular courses • Are there gaps in the curriculum allowing some students to miss regular attention to CT skills? • Use the results of assessments and analyses to suggest curricular modifications addressing problems revealed by the process

18. Our Guiding Principles:Faculty Input and Training • Workshops and retreats focused on defining and assessing CT • Discussion involvement and presentations by outside experts • Well paid faculty will participate in this project from start to finish • Improved quality control • Faculty buy-in regarding use of the rubric and the value of programmatic assessment

19. Our Guiding Principles:Including TMCC • Engaging TMCC Biology faculty in this sort of project should: • Improve our understanding of pedagogy at TMCC • Improve understanding by TMCC faculty of our curricular expectations • Facilitate the transition of transfer students into the UNR Biology major

20. Our Guiding Principles:Communicate Our Process and Outcomes to Others • Align efforts with the UNR Core Curriculum plans to assess general (not discipline-specific) CT skills • Inform Biologists at all other UCCSN institutions of our project, our rubric, and our results • Once our rubric is adopted by the department for use throughout the curriculum, it will be published on the departmental website • We will publish our process and analytical results in a journal specializing in educational research • E.g., The Journal of Research in Science Teaching