Peggy Maki Education Consultant Specializing in Assessing Student Learning

1. Taking a Backward-Designed Collaborative Approach to Assessing Student Learning to Explore both Students' Learning Processes and Products Peggy Maki Education Consultant Specializing in Assessing Student Learning Assessment Editor and Writer Keynote Presented at NSEE October 4, 2012

2. How well do your students… • Integrate • Transfer • Analyze • Apply or re-apply • Re-use • Synthesize • Restructure previous incorrect learning….

3. Within a course or module or learning experience? • Along the chronology of their studies and educational experiences? • From one subject or topic or focus or context to another one such as from an exercise to a case study or internship?

4. Percent Students Forget When They Log off or Leave Your Learning Experience

5. Foci • Research on Learning That Informs Teaching, Learning, and a Collaboratively Designed Chronological Approach to Assessing Student Learning • A Backward Designed Problem-based Approach to Assess Students’ Enduring Learning

6. Research on Learning That Informs Teaching, Learning, and Assessing Student Learning

7. Collaboration Technology Advising Service Learning Study Abroad Work Life Learner Internships Dorm Life Courses/ Modules Services Learning Communities • Knowledge is Socially Constructed (importance of peer-to-peer interaction)

8. Research on….. • Threshold Concepts: pathways central to the mastery of a subject or discipline that change the way students view a subject or discipline, prompting students to bring together various aspects of a subject that they heretofore did not view as related (Land, Meyer, Smith).

10. What’s The Problem in Physics? • Through the use of concept inventories, it is well documented that entry-level physics majors hold onto incorrect understanding of physics concepts—even ones that have been historically proven to be incorrect.

11. Despite all attempts to correct these incorrect concepts, such as incorporating more labs into early courses, students continue to draw on their incorrect understanding years after their early coursework often accounting for their weak performance as they progress in their studies or resulting in their dropping out of the major.

12. How to restructure incorrect understanding of physics concepts became the work of physics faculty at the University of Colorado (PhET project).

14. Learning Progressions: knowledge-based, web-like interrelated actions or behaviors or ways of thinking, transitioning, self-monitoring. May not be developed successfully in linear progression thus necessitate formative assessment along the trajectory of learning. Movements towards increased understanding (Hess).

15. Writing beyond what is visually presented during a lecture • Identifying clues to help organize information during a lecture • Evaluating notes after class • Reorganizing notes after class

16. Comparing note-taking methods with peers • Using one’s own words while reading to make notes • Evaluating one’s understanding while reading • Consolidating reading and lecture notes

17. Sharing practices on how to organize, think, and memorize content • Evaluating one’s own understanding • Monitoring the effectiveness of note- taking practices Source: Calvin Y. Yu, Director of Cook/Douglass Learning Center, Rutgers University

20. Approaches to Learning… • Surface Learning • Deep Learning

21. NRC, 2002

22. Cognitive Psychomotor Affective Forms of Representation within Contexts Integrated Learning….

23. Institution-level Outcomes (GE) Program- or Department-level Outcomes, including GE Course/Service Outcomes/Educational Opportunities Outcomes (including GE) Collaboratively Agreed upon Learning Outcomes at Multiple Levels Foster Enduring Learning

24. How Knowledge about Learning Shapes Teaching and Inquiry into Student Learning • What do you expect your students to demonstrate, represent, or produce by the end of their program of study or by the end of their undergraduate or graduate studies? • What chronological barriers or difficulties do students encounter as they learn--from the moment they matriculate? • How well do we identify and discuss those barriers and then track students’ abilities to overcome them so that “more” students achieve our expectations?

25. Mapping Multiple and Diverse Opportunities across Students’ Journey

26. A Backward Designed Problem-based Approach to Assessment That Is Grounded in Research or Study Questions…

28. Research or Study Questions

29. The Seeds of Research or Study Questions

30. A Taxonomy…. • Identifies patterns of weakness in student work or performance: analysis, support of claims, quantitative reasoning, etc. • Identifies context within which those obstacles occur. • With students, attempts to identify “why” students are struggling with those obstacles. • Chronologically alerts colleagues to weaknesses that still need to be addressed.

31. Some Examples of Research/Study Questions

35. Identification or Design of Assessment Methods That Provide Evidence of Product and Process

36. Direct Assessment Methods • Scenarios—such as online simulations • Critical incidents • Mind mapping • Questions, problems, prompts • Data analysis or data conversion

37. Problem with solution: Any other solutions? • Chronological use of case studies • Chronological use of muddy problems • Analysis of video • Smaller problems that ready students for internships, capstone projects, applied learning opportunities

38. Self-reflective journals or self-reflection on strengths and weaknesses in one’s own work leading to acceptance of the need to improve • Think aloud

39. Some Indirect Assessment Methods • Student Assessment of Their Learning Gains: SALG • Small Group Instructional Design: SGID • NSSE/CCSSE • Interviews/surveys

40. 1. Identify the outcome or outcomes you will assess. 2. State the Research or Study Question You Wish to Answer. 3. Conduct a Literature Review about that Question. 8. Share Developments within and outside the Institution to Build Knowledge about Educational Practices. 4. Develop a Plan to Collect Direct and Indirect Assessment Results that Will Answer Your Question. 7. Implement Agreed-upon Changes and Reassess. 6. Collaboratively Discuss Ways to Improve Pedagogy or Educational Practices 5. Analyze and Interpret Students’ Work and Students’ Responses.

41. Soft Times and Neutral Zones

42. Works Cited • Hess, K. 2008. Developing and Using Learning Progressions as a Schema for Measuring Progress. National Center for Assessment, 2008. http://www.nciea.org/publications/CCSSO2_KH08.pdf • Maki, P. 2010. 2nd Ed. Assessing for Learning: Building a Sustainable Commitment Across the Institution. VA: Stylus Publishing, LLC • Meyer, J.H., Land, R., and Baillie, C. 2010. Eds. Threshold Concepts and Transformational Learning. Rotterdam: Sense Publishers. • National Research Council. 2002. Knowing What Students Know: The Science and Design of Educational Assessment. Washington, D.C. • Yu, C. Y. “Learning Strategies Characteristic of Successful Students.” Maki, P. 2010. p. 139.