George D. Kuh Rankings and the Visibility of Quality Outcomes in the EHEA Dublin, Ireland

1. Measuring What Matters: Forging the Right Tools to Assess Student Learning Outcomes • George D. Kuh • Rankings and the Visibility of Quality Outcomes in the EHEA • Dublin, Ireland • January 31, 2013

2. Overview • The U.S. context • A word about NILOA • Assessment: Purposes and approaches • The kind of learning we need today • The measurement tools we need • Concluding thoughts

3. The U.S. Context • Unprecedented numbers of increasingly diverse students matriculating • Many underprepared students • Rising college costs • Continuing shift of cost from government to students • Increasing numbers of part-time instructors • Worries about university quality, global competitiveness

4. NILOA NILOA’s mission is to discover and disseminate effective use of assessment data to strengthen undergraduate education and support institutions in their assessment efforts. Surveys ● Web Scans ● Case Studies ● Focus Groups ● Occasional Papers ● Website ● Resources ● Newsletter ● Listserv ● Presentations ● Transparency Framework ● Featured Websites ● Accreditation Resources ● Assessment Event Calendar ● Assessment News ● Measuring Quality Inventory ● Policy Analysis ● Environmental Scan ● Degree Qualifications Profile www.learningoutcomesassessment.org

5. www.learningoutcomeassessment.org

6. Assessment 2013 • Greater emphasis on student learning outcomes and evidence that student performance measures up

7. Assessment 2013 • Greater emphasis on student learning outcomes and evidence that student performance measures up • Demands for comparative measures • Increased calls for transparency ---public disclosure of student and institutional performance • Assessment “technology” has improved markedly, but still is insufficient to document learning outcomes most institutions claim

8. Measuring Quality in Higher Education (Vic Borden & Brandi Kernel, 2010) Web-based inventory hosted by AIR of assessment resources. Key words can be used to search the four categories: • instruments (examinations, surveys, questionnaires, etc.); • software tools and platforms; • benchmarking systems and data resources; • projects, initiatives and services. http://applications.airweb.org/surveys/Default.aspx

9. Assessment Purposes • Improvement • Accountability

10. Two Paradigms of Assessment Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

11. Assessment Tools • Direct (outcomes) measures -- Evidence of what students have learned or can do • Indirect (process) measures -- Evidence of effective educational activity by students and institutions

12. Direct Measures • ETS Proficiency Profile & Major Field Tests • ACT Collegiate Assessment of Academic Proficiency (CAAP) • Collegiate Learning Assessment (CLA) – the AHELO measure of general skills • Competency tests (e.g., nursing, education) • Portfolios (authentic student work such as writing samples) • Performances, demonstrations

13. Indirect Measures • National Surveys of Student Engagement (NSSE/CCSSE/AUSSE/SASSE) • Beginning College Survey of Student Engagement (BCSSE) • Faculty Survey of Student Engagement (FSSE) • Cooperative Institutional Research Program (CIRP) • Your First College Year (YFCY) • College Student Experiences Questionnaire (CSEQ) • Noel Levitz Student Satisfaction Inventory

14. Institution-level assessments of learning outcomes for all institutions

15. Program-level assessments of learning outcomes for all institutions

16. What Really Matters in University: Student Engagement Because individual effort and involvement are the critical determinants of college impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student engagement. Pascarella & Terenzini, 2005, p. 602

17. The quality of student experiences varies more within than between institutions.

18. Supportive Campus Environment: 4th-Year Students at Master's Institutions Percentile 10 Percentile 50 Percentile 90 100 80 60 40 20 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Master's Institutions

19. % of Variance Between Institutions

20. Commensurate Complexity (Thorngate, 1976; Weick, 1985) Generally Applicable Simple Accurate

21. US Economy Defined by Greater Workplace Challenges and Dynamism • More than 1/3 of the entire US labor force changes jobs ANNUALLY. • Today's students will have 10-14 jobs by age 38. • Half of workers have been with their company less than 5 years. • Every year, more than 30 million Americans are working in jobs that did not exist in the previous year. DOL-BLS

22. The World Wants More From Us and Our Graduates • …more college-educated workers. • …more educated workers with higher levels of learning and knowledge.

23. Employer expectations of employees have increased % who agree with each statement Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past Employees are expected to work harder to coordinate with other departments than in the past The challenges employees face within our company are more complex today than they were in the past To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past

24. Key Capabilities Open the Door for Career Success and Earnings “Irrespective of college major or institutional selectivity, what matters to career success is students’ development of a broad set of cross-cutting capacities…” Anthony Carnevale, Georgetown University Center on Education and the Workforce

25. Narrow Learning is Not Enough: The Essential Learning Outcomes • Knowledge of Human Cultures and the Physical & Natural World • Intellectual and Practical Skills • Personal and Social Responsibility • “Deep” Integrative Learning

26. Deep, Integrative Learning • Attend to the underlying meaning of information as well as content • Integrate and synthesize different ideas, sources of information • Discern patterns in evidence or phenomena • Apply knowledge in different situations • View issues from multiple perspectives

27. Do we measure what we value?orDo we value what we measure? Wise decisions are needed about what and how to measure the proficiencies demanded by the 21st century

28. We need – and are poised for – a “sea change” in what counts as meaningful evidence of student progress and accomplishment.

29. Evidence of College Graduates Skills and Knowledge Very effective Fairly effective Supervised internship/community-based project 83% Senior project (e.g., thesis, project) 79% Essay tests 60% Electronic portfolio & faculty assessments 56% Multiple-choice tests 32%

30. To Get the Right Kind of Evidence… • We need assessment approaches sensitive to a wide variety of knowledge, abilities, proficiencies, and dispositions

32. Promising Experiments

33. Shift the national conversation from what is taught to what is learned by providing HEIs with a template of widely agreed-upon competencies required for the award of degrees. http://www.learningoutcomesassessment.org/DQPCorner.html

34. 3 levels: • Associate, Bachelor, Masters • 5 areas: • Broad, Integrative • Knowledge • Specialized • Knowledge • Intellectual Skills • Applied Learning • Civic Learning http://www.learningoutcomesassessment.org/DQPCorner.html

35. Occasional Paper #16 The Degree Qualifications Profile: Implications for Assessment Peter T. Ewell & Carol Geary Schneider This paper offers guidance for how to gather evidence about the extent to which the competencies described in the DQP are mastered at the levels claimed. The challenges associated with assessing DQP proficiencies are outlined. www.learningoutcomeassessment.org/OccasionalPapers.htm

36. Promising Experiments A dozen two- and four-year HEIs are using the VALUE Rubrics in a “proof of concept at scale” with an eye toward building a national vehicle for using common rubrics

37. Valid Assessment of Learning in Undergraduate Education (VALUE) Rubrics Inquiry and analysis Critical thinking Creative thinking Written communication Oral communication Reading Quantitative literacy Information literacy Teamwork Problem solving Civic knowledge and engagement Intercultural knowledge and competence Ethical reasoning and action Foundations and skills for lifelong learning Integrative learning

38. AAC&U VALUE Project – 15 Rubrics

39. Promising Experiments Massachusetts effort led by the Commissioner for Higher Education is enlisting additional states to use assessment of authentic student learning work to compare performance and monitor progress

40. Moving Quality Assurance Forward • Shift the motivation for assessment work from compliance mentality to institutional responsibility • Experiment with ways to “roll up” program level results into meaningful institution-level profiles of student accomplishment • Reconcile or ameliorate the tensions between the accountability and improvement purposes and uses of assessment

41. Moving Quality Assurance Forward • Show how assessment results are being used to modify curriculum and teaching and learning approaches and enhance student learning

42. Transparency

44. Voluntary System of Accountability(APLU/AASCU) www.collegeportraits.org

45. VSA Student Learning Outcomes Pilot • Four-year experiment • Value-added approach • Institutional level evidence • Administer and publicly post results: • Collegiate Assessment of Academic Proficiency • Collegiate Learning Assessment • ETS Proficiency Profile

46. Templates • Voluntary System of Accountability (APLU/AASCU) • U-CAN /Building Blocks for 2020 (NAICU) • College Navigator (NCES) • Transparency by Design/College Choices for Adults (WCET) • Voluntary Framework of Accountability (AACC) • Transparency Framework (NILOA)

47. Transparency Reports

48. Providing Evidence of Student Learning: A Framework for Transparency Based on an examination of about 1000 institutional websites, the Transparency Framework provides guidance to institutions for effectively presenting learning outcomes assessment information on their websites. Transparency Framework

49. http://planning.iupui.edu/assessment/

50. The things we have to learn before we do them, we learn by doing them. Aristotle, Nicomachean Ethics