Scientific Reasoning Competency Assessment in Higher Education in Virginia

1. Scientific Reasoning Competency Assessment in Higher Education in Virginia Harold Geller hgeller@gmu.edu Department of Physics and Astronomy George Mason University

2. What I’m Going to Talk About • SCHEV Mandate for Core Competencies • Scientific Reasoning Competency • Basis of GMU Response • Definition • Standards • Methodology • Analysis and Presentation • Commentary

3. SCHEV Mandate for Core Competencies • The State Council of Higher Education for Virginia (SCHEV) • Meeting on core competencies in November 2001 • See state issued guidelines • Original deadlines modified in 2002 • Responses required were delayed to 2003

4. Scientific Reasoning Competency • Not easily defined • One approach • Hazen and Trefil (1991) • Ability to comprehend science article in newspaper • Does this really define scientific reasoning? • Can this be left to the media? • Is this definition an end-all? • What grade level does this represent? • GMU approach • See GMU Assessment Proposal

5. Basis of GMU Response • Issues hashed out in committee • Definition • Implementation • Which students take exam • How many students take it • How administered • Ability to measure across disciplines • Lowest common denominator • Content independence

6. GMU Definition of Scientific Literacy[by committee] • Based upon 8 defined learning goals • Identify different scientific ways of knowing • Develop and test hypotheses • Interpret data • Recognize primary and secondary sources • Utilize qualitative and quantitative techniques • Recognize power and limitations of science process • Communicate among scientists and general public • Recognize role of science in decision making

7. GMU Standards[by committee] • To be set and established by committee consisting of • All faculty that teach general education science courses • Members of the Scientific Reasoning Competency Assessment Committee

8. GMU Methodology[by committee] • Develop common test items • Reflect goals in definition • Pilot questions • Use in final examinations of sample of general education science courses • Count as part of grade in class • Analyze results of pilot testing • Develop standards based upon results in sample of classes

9. As Part of Methodology(a syllabus addendum)[by committee] • Sample supplement to all general education science classes’ syllabi • “One of the goals of the general education program at George Mason University is to develop students competent in scientific reasoning. This course will help you achieve that goal. In order to judge university success in reaching this goal, a test will be given in late March or early April that will focus on scientific reasoning. This required multiple choice test will take approximately half an hour or so to complete and will count toward your grade.”

10. Sample Question (vetoed)[by Geller] • Johannes Kepler (1571-1630) derived his three Laws of Planetary Motion solely from the observations of Tycho Brahe (1546-1601). This represents which of the following ways of scientific knowing? • A inductive • B deductive • C empirical • D theoretical • E none of the above

11. Sample Question (vetoed)[by Geller] • The above graphs represent a phase diagram for water and carbon dioxide. Using the phase diagram for carbon dioxide, what phase would carbon dioxide be found at a pressure of 0.5 atmosphere and a temperature of -78.5o C? • A liquid • B gas • C solid • D any of the above is possible • E none of the above

12. Sample Question (vetoed)[by Geller] • The above graphs all represent experimental values observed over a period of time. The graph that best represents a value that is increasing exponentially over time is which one of the above graphs? • A Graph A • B Graph B • C Graph C • D Graph D • E None of the above

13. Sample Question (vetoed)[by Geller] • Stefan-Boltzmann's Law relates the energy output of an object to its temperature. The energy is proportional to the temperature (in Kelvin) to the fourth power. Therefore, if the temperature (in Kelvin) of an object is doubled, its energy output will increase by how many times? • A 2 • B 4 • C 8 • D 16 • E 32

14. Sample Questions (affirmed)[Article from Washington Post] Tumors Prevalent in Anacostia's Fish Rate Rivals Highest Recorded for U.S. River; Vehicle Pollution Blamed By Monte Reel Fish in the Anacostia River have cancerous tumor rates that are as high as ever documented in an American river, and a U.S. government-led study to be published next month links the tumors to pollution caused by vehicle emissions and runoff. Fifty to 68 percent of mature brown bullhead catfish collected in 2001 from three parts of the river in the city had liver tumors, most of which were cancerous, according to the study led by the U.S. Fish and Wildlife Service. In addition to the liver tumors, 13 to 23 percent of the bullheads had skin tumors, scientists found…

15. Sample Questions (affirmed)[based upon article from Washington Post] • Which one of the following graphs is consistent with the data presented in the article on tumors of Bullhead Catfish at a specific point in the Anacostia river? (for competency 3)

16. Sample Questions (affirmed)[based upon article from Washington Post] • Which of the following hypotheses are justified given the data in the article? You expect that fish who eat insects in the same river will: (competency 2) • Have the same level of tumors as brown bullhead catfish because they are living in the same environment. • Have more tumors because the insects also pick up the air pollution. • Have fewer tumors than bullhead catfish because they do not eat from the river bottom. • Can’t be predicted because what the fish eat does not matter.

17. GMU Analysis and Presentation[by committee] • Present aggregate data to SCHEV • Percentages of high proficiency • Percentages of acceptable proficiency • Percentages of low proficiency • Adjust as needed • Determination to be made in pilot testing

18. Commentary[by Geller] • How best to test scientific literacy (a.k.a. assessment) • Essays, multiple choice, short answer, mixes • Problems with need to apply across disciplines • Lowest common denominator • Content independence • How important is content? • Comparison with state examinations • Standards of Learning (Virginia) • Comparison with reading comprehension • New York Times is aimed to be 9th grade reading level • Washington Post aims at 8th grade reading level

19. Commentary[by Geller] • Value of tests • Measure of higher education or lower education? • Added burden to all lower level science classes • What if students take more than one of these introductory science courses? • Some students take astronomy as general elective. • This can skew statistics. • Unfunded mandates • More attention to something other than the course itself? • Why is this a college/university requirement? • Failure of “lower education”?

20. Acknowledgements • GMU Office of Institutional Assessment • Karen Gentemann • Ying Zhou • Center for Teaching Excellence • Laurie Fathe • GMU Scientific Literacy Competency Committee • Department representatives • Faculty volunteers • Department Colleagues • Robert Ehrlich, Robert Ellsworth, Maria Dworzecka, Greg Foster (Chemistry), Joe Weingartner