What Are Students Learning In Your Course? Strategies for Assessment

1. What Are Students Learning In Your Course? Strategies for Assessment Karl Wirth Macalester College Karen Viskupic Boise State University Earth Educators’ Rendezvous 22 July 2016

2. How Do We Know…? • Students graduating with a geology degree should: • recognize common rocks and minerals in hand sample • Introduce yourself to a neighbor • How could you assess this goal?

3. Goals for Today’s Workshop • Workshop participants will: • be able to articulate clear and assessable outcomes for student learning; • know how to construct and implement a knowledge survey in their course; • be able to explain the goals and design of exam wrappers; • know how to write graduated test questions that span different levels of thinking and learning.

4. Why Do We Assess?

5. Purposes of Assessment • design effective learning experiences • track student progress • focus student effort • provide feedback for improved learning • improve instruction • assign grades • improvement of programs and curricula • personnel review • accreditation of institutions and programs • design effective learning experiences • track student progress • focus student effort • provide feedback for improved learning • improve instruction • assign grades • improvement of programs and curricula • personnel review • accreditation of institutions and programs

6. Formative vs Evaluative Assessment When the cook tastes the soup, it is formative; when the guest tastes the soup it is evaluative R. Stake (cited in Earl, 2004)

7. What Do We Assess?

8. Knowledge Dimension • Factual Knowledge • Conceptual Knowledge • Procedural Knowledge • Metacognitive Knowledge

9. Knowledge Dimension Factual Knowledge • Terminology • Specific details and elements Conceptual Knowledge • Classifications and categories • Principles and generalizations • Theories, models and structures Procedural Knowledge • Subject-specific skills and algorithms • Subject-specific techniques and methods • Criteria for determining when to use procedures Metacognitive Knowledge • Strategic knowledge • Cognitive tasks, incl. context and conditional knowledge • Self-knowledge

10. Critical Thinking • Communication Skills • Spatial Visualization Skills • Scientific Research Methods • Managing Uncertainty • Ability to Integrate and Continue to Learn • Quantitative Skills • Field Skills • Teamwork and Working With Diversity • Working with Data and Technology Geoscience Skills Mosher (2015)

11. Goals for Learning and Assessing

12. Approaches to Course Design “Traditional” “Backward” Enduring Understandings and Learning Goals Acceptable Evidence Instructional Activities List Course Topics Design Instruction Prepare Exams/Papers Wiggins and McTighe (1998)

13. Appropriate Assessments • Traditional quizzes & tests • selected response • constructed response • Performance tasks • open-ended • complex • authentic Worth being familiar with Important to know and do Big Ideas & Enduring Understanding Wiggins and McTighe (1998)

14. Example Goals • Students graduating with a degree should: • Recognize common rocks and minerals in hand sample and thin section. • Read topographic and geological maps and airphotos. • Make and analyze geological and geophysical measurement and interpret geological features in the field • Understand the origin, structure and history of the Earth and how the Earth System works

15. Goals vs. Outcomes Example

16. Write Learning Outcomes • For one learning goal in your course • Write several learning outcomes “By the end of this course, students will be able to…” Describe learning within a context Rely on active verbs Can be assessed quantitatively or qualitatively

17. Bloom’s Cognitive Processes

18. Formative Assessment: Knowledge Surveys

19. Example Knowledge Survey I am unable to perform the task I am able to partially perform the task I can perform the task for evaluation

20. Knowledge Survey Results Pre-Learning

21. Knowledge Surveys • Introduced by Nuhfer (1993, 1996) • Knowledge & skill outcomes of course • Span levels of thinking (Bloom levels) • Students self-report knowledge/ability

22. Example Survey Items

23. Knowledge Survey Results

24. Knowledge Survey & Exam I

25. Knowledge Survey & Exam II

26. Self-Assessment Skill Nuhfer et al. (2016)

27. “Map” of Learning for Dynamic Earth Pre-Course Survey Students Survey Items

28. “Map” of Learning for Dynamic Earth Pre-Course Survey Pre-Exam I Survey

29. “Map” of Learning for Dynamic Earth Pre-Course Survey Pre-Exam I Survey Pre-Exam II Survey

30. “Map” of Learning for Dynamic Earth Pre-Course Survey Pre-Exam I Survey Pre-Exam II Survey Pre-Exam III Survey

31. Interpreting Knowledge Surveys • What is shown? • What does it mean? • What sorts of interventions or changes could you (the instructor) make to improve learning?

32. Results of pre-course knowledge survey, pre-exam I and II knowledge surveys, and exam I and II scores.

33. Results of pre-course (orange) and pre-exam I (blue) knowledge surveys. Full scale = “know answer.”

34. Average responses to knowledge survey questions before (2000-2003 courses) and after (2005) intervention. Interventions were made to the units on crystallography and x-ray mineralogy in 2004-2005. (from Dexter Perkins)

35. Bar chart showing Bloom levels of survey items for different units of a course.

36. Knowledge Survey Results Learning Gains Post-Learning Pre-Learning

37. Utility of Knowledge Surveys Course Design Clarification of course objectives and structure Improved organization and preparation Students Full disclosure of course objectives and expectations Study guide Formative assessment tool Development of self-assessment skills Instructors Assessment of learning gains Course assessment Assessment of instructional practices Programs Program Objectives Student Learning

38. Reflecting on Feedback: Exam Wrappers

39. Exam “Wrapper” • Reflection • Preparation Strategies • Performance Analysis • Planning • Revision • Exam Re-do Achacoso (2004) Lovett (2008)

40. Exam I Revisions

41. Exam “Wrapper” Results Study Strategies Analysis of Errors

42. Exam Preparation

43. Learning Wrapper • Select a feedback opportunity (e.g., graded exam; comments on term paper) in one of your courses • Consider how you would like students to use and reflect on that feedback • Outline a “wrapper” to facilitate improved learning and reflection

44. Assessing for Expertise: Graduated Tasks

45. Next Generation Science Standards “This integrated perspective of what it means to know science suggests that assessment should help determine where a student can be placed along a sequence of progressively more ‘scientific’ understanding of a given core idea that by definition includes successively more sophisticated applications of practices and cross-cutting concepts” Pellegrino (2013), Science

46. PISA Framework “The functional use of knowledge requires the application of those processes that are characteristic of science and scientific inquiry” “test questions (items) require the use of the scientific competencies within a context. This involves the application of scientific knowledge.” • Knowledge about science • Scientific competencies Pisa (2009) Assessment Framework

47. Task Example - Tectonics Objective: Students will recognize essential plate tectonic features and the evidence that supports the theory Know: Know a definition of a plate boundary Understand: The characteristics of important tectonic features and processes Do: Use global maps of topographic, seismic, seafloor age, and volcanic features to identify and locate active plate boundaries

48. Tectonics Performance Task • Level “Know” • Which of these locations is a plate boundary? Circle one. • Border between continental to oceanic crust • Coastline • Abyssal plain • Mid-ocean ridge • Hot spot

49. A Tectonics Task (cont’d) B C Level “Understand” Match the letter on the map to the correct feature listed below. E D ____ Highest heat flow ____ Continental shelf ____ Possible hot-spot ____ Oldest sea floor ____ Fracture zone F

50. A Tectonics Task (cont’d) B C Level “Understand” Predict the motion of location D relative to the mid-ocean ridge on this map and provide two observations or reasons that support your hypothesis. E D F