Interpreting Data & “Closing the Loop”

1. Interpreting Data & “Closing the Loop” Margaret Kasimatis, PhD VP for Academic Planning & Effectiveness

2. Location in Assessment Cycle DETERMINE PRACTICES USED TO ACHIEVE OUTCOMES IDENTIFY SPECIFIC OUTCOMES GATHER EVIDENCE ARTICULATE MISSION/ GOALS RECOMMEND ACTIONS REVIEW & INTERPRET RESULTS

3. Step #1: Organize Your Data • Assemble the following in one place: • All data or possible sources of data • List of learning outcomes/research questions • Curriculum, experiences, practices • Map data sources to outcomes • Outcomes x Measures Map • Map results to outcomes • Present in “user-friendly” way

4. Mapping Data Sources to Outcomes: Engineering Example

5. Presenting Results • It’s best to present results one outcome at a time • It’s not recommended to present results primarily by measure • Make sure to present the data in a user-friendly way (i.e., in a way that faculty in your department are comfortable with)

8. Step #2: Interpret the Data • What type of criterion? • What is “significant?” • Are the findings reliable/valid?

9. What type of criterion? • Norm-referenced • Need appropriate comparison group • Avoid percentile rankings • Good for more nebulous findings • Absolute standard • Usually more appropriate for performance-based measures

10. What is significant? • Important to test when making comparisons • Numbers that look different may not really be different • However, just because a difference is significant doesn’t mean it’s important

11. Are findings reliable/valid? • Can we believe student self-reports? • Are standardized measures more valid? • Single measure vs. multiple measures

12. Step #3: Make Recommendations • Start by considering • Where in the curriculum are outcomes addressed? And at what level? • Refer to (or create) curriculum map, or • Inventory for particular outcome • How? What practices/techniques are used? • Where are the gaps? • What can be changed? • In the short-term? In the long-term?

13. Step #3: Make Recommendations • Possible types of recommendations: • Changes to pedagogy • Changes to curriculum/programming • Allocation of resources • Keep in mind that you can’t fix everything at once – so start small • It’s usually better to modify than add • Be as specific/concrete as possible

14. Step #4: Implement & Document Recommend Actions Example

15. Example of Assessment Loop • Step #1: Articulate Goals/Outcomes • Goal: Graduates will possess strong communication skills • Outcome: Graduates will be able to write clearly and effectively • Step #2: Determine practices used to achieve outcome • Survey of required writing experiences in Core and academic programs

17. Example of Assessment Loop, cont’d • Step #3: Conduct measurements • Higher Education Research Institute (HERI) College Student Survey (CSS) • National Survey of Student Engagement • HMC Alumni Survey

18. Example of Assessment Loop, cont’d • Step #4: Review & interpret results • Faculty committees • Assessment • Curriculum • Teaching & Learning • Full faculty - workshop

19. Writing SkillsFrequency data from HERI, NSSE & Alumni Survey Liberal Arts 4-Yr Non- Sectarian 1959-1987 Cohort 1988-2001 Cohort AITU Seniors HMC Seniors HMC Seniors

20. Seniors’ Perceptions of Writing Skills by Major (from NSSE)

21. Alumni Perceptions of Writing Emphasis by Major(alumni graduating in last 10 years)

23. Example of Assessment Loop, cont’d • Step #5: Recommend actions • Universal writing handbook • Faculty workshop on writing instruction • Revision to Core Curriculum

24. MULTIPLE CURRICULAR OBJECTIVESIN HORIZONTALLY INTEGRATEDCORE CURRICULUM

25. QUESTIONS?