Assessing GE Goals and Capacities with Concept Inventories: One Path Forward . . . Part I

1. Assessing GE Goals and Capacities with Concept Inventories:One Path Forward . . . Part I • Overview: General Education • Overview: Concept Inventories • Confluence: SLCI Results from S13 GE Natural Sciences Courses • Confluence: Math Concept Inventories for GE Quantitative Reasoning • Moving Forward: Next steps and segue into Part II Assessment takes a village . . . Catherin Atkins, College of Science Janet Bowers, Mathematics Geoff Chase, Undergraduate Studies Douglas Deutschman, Biology Reynaldo Monzon, STAR Chris Rasmussen, Mathematics Stephen Schellenberg, Geological Science Kathy Williams, CTL . . . with thanks to S13 GE Nat. Sci. Facultyand Ed Nuhfer of Humboldt State Fun facts: ~1/3 of BS/BA is GE WASC is interested in GE GE is integral part of mission

2. Goals and Capacities (G&Cs) of SDSU General Education Program • Goals for Natural Sciences (Three Foundation, One Exploration) • Explain basic concepts and theories of the natural sciences • Use logic and scientific methods to analyze the natural world and solve problems • Argue from multiple perspectives about issues in natural sciences that have personal and global relevance • Use technology in laboratory and field situations to connect concepts and theories with real-world phenomena Are these G&Cs being assessed at thecourse level?If so, how? How can these G&Cs be introduced, developed, and demonstrated within and across courses? How can we assessthese G&Cs at a program level? • Goals for Quantitative Reasoning (One Foundation) • Apply appropriate computational skills and use basic mathematical concept to analyzed problems in the natural and social sciences • Use methods of quantitative reasoning to solve and communicate answers to real-world problems • Essential Capacities Developed through General Education • Construct, analyze, and communicate arguments • Apply theoretical models to the real world • Contextualized phenomena • Negotiate differences • Integrate global and local perspectives • Illustrate relevance of concepts across boundaries • Evaluate consequences of actions

3. One Path Forward for Programmatic Assessment andCurricular Revision is the Concept Inventory Definition:Collection of questions designed to assess student understanding of thefoundational knowledge, concepts, and procedures for a given topic, discipline, etc. • Goals: • Assess scientific habits of mind and literacy • Identify gaps in approach understanding • Inform curricular reform Inform on multiple levels in multiple ways when linked with demographics: From synoptic overview of student population and needs toinsights from distribution of distractor responses Concept inventories have traditionally been discipline-focused,but recent efforts include metadisciplinary assessment of science literacy We hypothesize that such concept inventories provide a means for programmatic assessment and curricular revision that will serve our students, faculty, and institution

4. Two Example Questions from 25-Item Science Literacy Concept Inventory(SLCI of Nuhfer et al., pers. comm.) GE Capacities: 1 2 3 4 5 6 7 GE Goals (Nat. Sci.): 1 2 3 4 6. To help us to understand the lunar phases, we have set up a basketball, a baseball, and a golf ball to represent respectively the Sun, Earth and the moon. What method of science are we employing? A. Experiment. Moving the balls can allow us to measure the size of the shadow that one ball casts on another ball. B. Modeling. Moving the balls helps us to perceive the positions of the celestial bodies that might explain the observed phases. C. Multiple working hypotheses. Moving the balls can allow us to determine whether the lunar phases were different during the ice ages. D. None. Moving the balls differs from reality to such an extent that it is an ineffective way to understand lunar phases. 24. Which of the following assumptions is important to all sciences? A. Humans can understand the physical world through laws they can discover. B. The experimental method is the only valid way to test hypotheses. C. Life is not governed by the same physical laws as non-living systems. D. Random events have no role in the actual physical world. GE Capacities: 1 2 3 4 5 6 7 GE Goals (Nat. Sci.): 1 2 3 4

5. Summary of Results from SCLI Deployment across Suite ofGE Foundation and Exploration Courses during early Spring 2013 2,794 InitialRecords Non-Response Bias: Female 25.2% Male 39.1% GEOL303 9.2% BIOL100 18.3% BIOL326 43.3%ASTR310 56.4% SLCI? 870(31%) No Yes SAT/ACT Not Reported: Native 3.8% Transfer 60.8% Taking Foundations: Native 71.6% Transfer 31.4% Native? 548(20%) No Yes First Generation: CHEM100 29.3% All Others 19.0% Minority: CHEM100 73.7% All Others 54.3% CHEM100? 216(8%) Yes No 67(2%) Missing Data Yes No 1,093 Retained Records (39%)

6. Overall SLCI Performance and Potential Explanatory Variables

7. Correlations of Explanatory Variables with SLCI Performance R2 = 20.5%

8. Correlations of SAT (ACT) and GPA with SLCI Performance R2 = 12.6% R2 = 6.3%

9. Aside: Snapshot of when Foundation and Exploration courses are being takenby our students . . . Implications forscaffolding, advising, anddegree progress?

10. Foundations Patterns, trends, and explanatory hypotheses? R2 = 15.0% R2 = 7.0% Explorations R2 = 6.0% R2 = 5.2%

11. SLCI Performance Across Courses

12. End-Member Comparison Performance difference cannot be attributed to incoming SAT scores Performance difference may be partially explained by differences in age, major, units earned, and other confounded variables

13. End-Member Comparison End-Member Comparison Performance difference cannot be attributed to incoming SAT scores Performance difference may be partially explained by differences in age, major, units earned, and other confounded variables

14. End-Member Comparison End-Member Comparison Performance difference cannot be attributed to incoming SAT scores Performance difference may be partially explained by differences in age, major, units earned, and other confounded variables

15. End-Member Comparison End-Member Comparison Performance difference cannot be attributed to incoming SAT scores Performance difference may be partially explained by differences in age, major, units earned, and other confounded variables Majority of the R2 is not uniquelyassignable since it is shared amongrelated variables (i.e., confounded)

16. A Working Model . . . Limited by the Nature of the Current Data SLCI Next Steps: Deploy more broadly over multiple semesters to allow focuson science literacy gains through time within individuals

17. Alternate or Parallel Concept Inventory? Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012)

18. Alternate or Parallel Path? Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012)

19. Alternate or Parallel Path for GE Natural Sciences? Test of Scientific Literacy Skills (TOSLS) – Gormally et al. (2012) Ongoing Efforts: Programmatic: Mapping Concept Inventory skills onto GE G&Cs Administrative: Implementing deployment across GE Nat. Sci. courses Grass-Roots: Using course content to introduce, practice, and master GE G&C

20. From GE Natural Science to GE Quantitative Reasoning: Fall 2012 Calculus Concept Inventory • Goals: • Assess scientific habits of mind and literacy • Identify gaps in approach understanding • Inform curricular reform FLIPPED FLIPPED TRADITIONAL TRADITIONAL Operate and inform on multiple levels in multiple ways: Synoptic overview of student population toinsights from distribution of distractor responses TECH TECH EXPERIENCED EXPERIENCED September 2012 Pre-Test December 2012 Post Test

21. Pre-Post Test Gains: Fall 2012 Calculus Concept Inventory

22. Example Question 16. The drawing represents a loaf of bread with a slice shown x inches from the left-hand end of the bread. Which of the following graphs could represent the volume V of the bread to the left of the slice as a function of the distance x from the left-hand end of the slice?

23. Operate and inform on multiple levels in multiple ways: Synoptic overview of student population

24. Assessing GE Goals and Capacities with Concept Inventories:One Path Forward . . . Part I • Overview: General Education • Overview: Concept Inventories • Confluence: SLCI Results from S13 GE Natural Sciences Courses • Confluence: Math Concept Inventories for GE Quantitative Reasoning • Moving Forward: Next steps and segue into Part II Assessment takes a village . . . Catherin Atkins, College of Science Janet Bowers, Mathematics Geoff Chase, Undergraduate Studies Douglas Deutschman, Biology Reynaldo Monzon, STAR Chris Rasmussen, Mathematics Stephen Schellenberg, Geological Science Kathy Williams, CTL . . . with thanks to S13 GE Nat. Sci. Facultyand Ed Nuhfer of Humboldt State Fun facts: ~1/3 of BS/BA is GE WASC is interested in GE GE is integral part of mission