Framing Evaluation of Complex Educational Intervention through Systems Thinking: InTeGrate Case Study

1. Using systems thinking to frame the evaluation of a complex educational intervention: InTeGrate as a case study Kim Kastens, Carol Baldassari, Jacqueline DeLisiand CathrynManduca American Geophysical Union December 17, 2014

2. What is InTeGrate? Elements of systems thinking to be discussed Using systems thinking to design, implement and evaluate InTeGrate

3. InTeGrate: Interdisciplinary Teaching of Geoscience for a Sustainable Future • NSF STEP Center = STEM Talent Expansion Program • Undergraduate education • $10M 5years finishing year 3 • Two required goals: • National impact on increasing number of students in STEM pipeline • Address a national grand challenge: in InTeGrate’s case, environmental sustainability and resource limitations.

4. Materials development: • Modules or full courses • Developed by 3-4 person faculty teams • Each team member must be from a different institution • Each team member must commit to testing the material • 36 teams • 109 team members, 23 states • Two year process • 6 published • 9 tested in 35 courses with 1338 students

5. InTeGrate Curriculum Development & Revision Rubric • Guiding principles (non-negotiable): • Grand challenge facing society • Interdisciplinary problem solving • Geoscientific habits of mind • Authentic geoscience data • Systems thinking • Pedagogical excellence: • Learning objectives & goals • Assessment & measurement • Resources & materials • Learning strategies • Alignment of module/course elements

6. Workshops: • 13 in-gathering workshops completed • 276 participants • Switching now to dissemination workshops • Website: • 1038 public-facing pages • 35,000 visitors in last year • serc.carleton.edu/integrate

7. Implementation Programs: • Larger scale than a single course • Innovative approaches to increase number and diversity of students developing Earth literacy OR • Prepare diverse workforce equipped to bring geoscience to bear in addressing societal issues • 7 in place, goal 26

8. Complex system

9. Elements of systems thinking we are using Systems mapping Non-linear causal chains Constant “sensing” of the system and its environment for decision-support Emergent phenomena Feedback loops Plastic behavior: lasting change

10. Systems mapping(related to conjecture mapping and logic modeling) Inputs Activities Outputs Outcomes ...leads to … …favors… …influences… …contributes to … …shapes the planning for… For each link: • Is there a mechanism? • Is there empirical evidence?

11. Conjecture mapping of InTeGrate January 2012 For each link: • Is there a mechanism? • Is there empirical evidence?

12. Elements of systems thinking Systems mapping Non-linear causal chains Constant “sensing” of the system and its environment for decision-support Emergent phenomena Feedback loops Plastic behavior: lasting change

13. x Cause Effect Action Nudge Effect Influence Action Nudge

14. x Intervention Outcome Outcome 1 Outcome 2 Activity Outcome 3

15. InTeGrate’s design conjectures that any given desirable outcome will require multiple nudges or influencers. Frequent intense discourse with teammates around learning goals, content, pedagogy Leads to Individual and group reflection on what is and is not working Leads to Instructors incorporate InTeGrate values and methods into the rest of their teaching practice Instructors who have internalized InTeGrate values and methods Leads to Frequent immersion in rubric, web site and other guiding materials Leads to Substantial interactions with assessment team consultant (yr 1) and development team leader Advocates for InTeGrate’s ideas, programs and materials become Leads to forge strong collaborative ties that can grow into Enduring community of practice

16. Evaluation inserts probes at critical flows, seeking evidence for the veracity and robustness of the conjectured flow. “Having goals and objectives formalized forced me to think differently about the design of course. …InTeGrate’sprocess forces you to think about how to start, where are you going towards. … I’m thinking about redoing all my classes.” Instructors incorporate InTeGrate values and methods into the rest of their teaching practice Instructors who have internalized InTeGrate values and methods Leads to Evaluation target Advocates for InTeGrate’s ideas, programs and materials become forge strong collaborative ties that can grow into Enduring community of practice

17. Elements of systems thinking Systems mapping Non-linear causal chains Continuous “sensing” of the system and its environment for decision-support Emergent phenomena Feedback loops Plastic behavior: lasting change

18. Complex problems require constant “sensing” of the status of the system and its environment… Meadows, D. (1999). Leverage Points: Places to Intervene in a Systems: The Sustainability Institute.

19. Use of technology to support decision-making http://faculty.cs.byu.edu/~mike/mikeg/WORKSHOP/panel.html http://serc.carleton.edu/admin/review/process_overview.php?process_id=3

20. Elements of systems thinking Systems mapping Non-linear causal chains Constant “sensing” of the system and its environment for decision-support Emergent phenomena Feedback loops Plastic behavior: lasting change

21. Emergent phenomena Towards STEM careers http://www.pbs.org/wgbh/nova/sciencenow/3410/03-ever-01.html Larger entities, patterns, and regularities arise through interactions among smaller or simpler entities that themselves do not exhibit such properties.

23. (GLE & embedded assessments) Non-Geo majors with increased understanding of Earth and environment (aka increased “geoliteracy”) Attitudinal survey; GLE essay questions Teach Geo modules and courses that are pedagogically excellent and oriented towards societally important Earth-related problems Progress on nationally- urgent problems concerning resources and environment More Geo graduates who are equipped and disposed to address societal problems of resources and environment Attitudinal survey More students attracted to Geoscience majors and careers STEM (Geo) Talent Expansion Inputs Activities Outputs Outcomes

24. Elements of systems thinking Systems mapping Non-linear causal chains Continuous “sensing” of the system and its environment for decision-support Emergent phenomena Feedback loops Plastic behavior: lasting change

25. Feedback loops causes or influences (B)…. …and then (B) in turn causes or influences (A). http://www.smashingmagazine.com/2013/02/15/designing-great-feedback-loops/

26. Balancing feedback loops: Materials development teams are required to have 3 or 4 people, from different types of institutions, and all developers must teach and test materials. Materials are being developed to be suitable for a wide range of students and institutions Materials suitable for a wide range of students and institutions Materials are widely adopted/ adapted Note that feedback loop does not pass through the leadership team… ... Individuals throughout the InTeGrate ecosystem are empowered to detect and act on opportunities and fix problems. (-) One instructor/ developer proposes activity optimized for his/her locale, institution, students Another instructor/ developer objects The group compromises (-) Module is revised to suit wider range of students & settings. Narrowly suitable activity works sub-optimally in other test schools Narrowly suitable activity is included in the test module

27. Elements of systems thinking Systems mapping Non-linear causal chains Continuous “sensing” of the system and its environment for decision-support Emergent phenomena Feedback loops Plastic behavior: lasting change

28. As practitioners of an historical science, geoscientists know that events leave traces, even long after the causal forces have ended. What will be the traces left behind on the landscape of higher education when the InTeGrate event has ended?

29. Potential types of traces include. . . Shared vision: of education, of geosciences Tested, reusable materials and tools Effective, transferable processes People who do things differently: faculty, students, graduates Relationships among people: Friendships, collegial relationships, and trust

30. Shared vision: The field of geosciences offers insights and approaches that are fundamental to tackling the 21st century’s grand challenges… From a non-geoscientist: “I think the materials did an excellent job of acquainting the students with the meaning of environmental justice. The materials also taught them how important science is to social justice.” From a geoscientist: “I always had this sense that these grand challenges were a better way to teach the basic stuff. InTeGrate’s structure, their guiding principles, affirmed that idea. …”

31. Tested & effective tool: Curriculum Development & Revision Rubric “The rubric really helped me focus my materials, resources, and activities on geoscientific habits of mind ….” “The rubric pushed us in directions that we might not have gone—the interaction between earth science and economic, societal, and policy issues.”

32. Effective, transferable management processes: Use of data and technology to create a decision-support system

33. People who do things differently:Graduates who make different life choices ….most accurately depicts your level of interest in a career in Earth or Environmental Sciences…. 36%

34. Relationships among people: Friendships, collegial relationships, and trust Enduring community of practice Fosters strong collaborative ties that can grow into “I’m really happy to have this opportunity to be involved in the project. I was inspired to work with great colleagues, share new ideas, and be involved in a new paradigm in curriculum development. I learned a lot and I would love to do a project like this again.”

35. Recap • Education is a complex system • Systems thinking (or complexity theory) is a well developed discipline with a useful toolkit for thinking about education reform • Systems mapping • Non-linear causal chains • Continuous “sensing” of the system for decision-support • Emergent phenomena • Feedback loops • Plastic behavior: lasting change • InTeGrate is an ambitious work in progress. Early indicators are that it may succeed in leaving lasting traces upon the landscape. • Shared vision • Tested, reusable materials and tools • Effective, transferable processes • People who do things differently: faculty, students, graduates • Relationships among people: Friendships, collegial relationships, and trust

36. Additional thinking from: • Anne Egger • Ellen Iverson • David McConnell • David Steer • InTeGrate Advisory Board • InTeGrate Leadership Team • Literature insights from: • Donella Meadows • Margaret Hargreaves • William Sandoval • John Kania, Mark Kramer & Patty Russell • HalliePreskill & SrikGopal • Michael Quinn Patton ?