The Landscape Changes: Assessment, Accreditation, and Accountability

1. The Landscape Changes: Assessment, Accreditation, and Accountability Robert Mundhenk and Jennifer Fager Assessment of Academic Achievement Conference University of Wisconsin-Superior 18 January 2011

2. Three Views of AssessmentView 1: • Assessment is a process in which those responsible for student learning on campus collect, analyze, and use information about student learning to improve it.

3. Three Views of Assessment:View 2 • “Outcomes-assessment practices in higher education are grotesque, unintentional parodies of both social science and ‘accountability.’ No matter how much they purport to be about ‘standards’ or student ‘needs,’ they are in fact scams run by bloodless bureaucrats who . . . do not understand the holistic nature of a good college education.” --Laurie Fendrich, “A Pedagogical Straitjacket,” Chronicle of Higher Education, June 8, 2007, B6.

4. Three Views of Assessment:View 3: • “The real culprit is the education establishment’s continuing reluctance to give the public a clear and accurate glimpse at how colleges spend their money. What’s missing, by and large, is ‘sunshine.’” --John A. Boehner, Speaker of the House of Representatives, though said as Chair, House Committee on Education and the Workforce

5. Three Conflicting Forces: • Assessment to Improve Learning • Assessment as Meaningless Bureaucratic Imposition that Violates Academic Freedom • Assessment as an Element of Institutional Accountablity

6. The National Scene: • [Colleges constantly tell us] “Everything is fine, we are doing a great job, send us more money, and leave us alone.” --Rep. Buck McKeon, House Committee on Education and the Workforce

7. The New Accountability • SPELLINGS COMMISSION (8/3/06): • “ . . . The quality of student learning at U.S. colleges and universities is inadequate and, in some cases, declining.” (7) • “Accreditation reviews are typically kept private, and those that are made public still focus more on process reviews than bottom-line results for learning or costs.” (13)

8. The New Accountability • Duncan Department of Education attacks the laxity of accreditors, including HLC, in assuring institutional and program quality • “Gainful employment” and “program integrity” assurance pushed by Duncan Department of Education • Student debt levels and employment rates

9. The New Accountability • “But can I assume that my graduate is going to be literate, know how to write, be exposed to math, be taught by senior faculty?” --Anne D. Neal, National Advisory Council on Institutional Quality and Integrity, 12/20/07

10. The New Accountability • “Institutions are asked, ‘Are you measuring student learning?’ And they check yes or no. That must change. Whether students are learning is not a yes-or-no question. It’s How? How much? And to what effect?” --Margaret Spellings Sept 06

11. The Newest Accountability • "We think -- I certainly think -- that it would be helpful when students head off to college that they have some basic information that they often don't have now: what's this going to cost, what's the [loan] repayment at that school, the expected graduation rate, the placement rate if I go into certain fields. . . .

12. The Newest Accountability • “One fallout from the gainful employment discussion is that a lot of the things people are pushing the for-profits for would be useful for all students to have, whether they're going to a land-grant or a private college, to a for-profit or a nonprofit. I think we'll be likely to look at getting students that kind of information. John Kline (R-MN), Chair, House Committee on Education and Labor (IHE 12/23/10)

13. Spellings Commission Recommendations (8/9/06) • “Robust culture of accountability” with a “consumer-friendly database that provides access to institutional performance and aggregate student outcomes” (21) • “Accreditation agencies should make performance outcomes, including completion rates and student learning, the core of their assessment . . . to . . . allow comparisons among institutions regarding learning outcomes and other performance measures” (23) • “Develop interoperable outcomes-focused accountability systems designed to be useful to students, policymakers, and the public. . . .”(22)

14. A Testing Service Response • Educational Testing Service: • “What is needed is a systematic, data-driven, comprehensive approach to understanding the quality of two-year and four-year postsecondary education, with direct, valid, and reliable measures of student learning.” • Workplace readiness and general skills • Domain-specific knowledge and skills • Soft skills . . . . • Student engagement with learning

15. Higher Education’s Response:APLU: “Voluntary System of Accountability” • Standard IPEDS data • Enrollment, persistence, graduation, etc. • Student “experiences and perceptions” • satisfaction and engagement data • Student learning outcomes: • Critical thinking • Analytic reasoning • Written communication • “. . . Using one of three tests”

16. The Boeing Approach • Ranking colleges based on graduates’ performance evaluations • “It’s really about improving the dialogue on curriculum, performance, and how we can build a stronger relationship between the colleges, universities, and us, because, ultimately, their students become our employees.” --Richard D. Stephens, Senior Vice President for Human Resources (Chronicle of Higher Education, 9/19/08)

17. The Bologna Approach • “Accountability in higher education begins with the establishment of public definitions of degrees and criterion-referenced statements of academic performance so that, when an institution awards a credential it can assert, with confidence, ‘This is what a degree represents, this is what the student did to earn the degree, and a warrantee has been issued on behalf of both institution and student.’ --Clifford Adelman, Learning Accountability from Bologna (IHEP, July 2008)

18. In Short: • Government and employers are seeking greater accountability from higher education • Government is demanding greater accountability and rigor of accrediting commissions • Higher education associations are responding with outcome-based models • Some approaches are clearly inadequate • Institutions must deal with internal blowback while dealing with external pressures

19. The Real Question: • How do we avoid being reductive (or worse) while accommodating the great triad of accountability, accreditation, and assessment?

20. The New Accreditation: Higher Learning Commission • Refocus on student learning within Five Criteria: • Mission and Integrity • Engagement and Service • Acquisition, Discovery, Application of Knowledge • Student Learning and Effective Teaching • Preparation for Future • The new Criteria see learning as infused throughout all Criteria, so responsibility for supporting, enhancing, and improving learning becomes an institutional responsibility

21. THE NEW PARADIGM • Shift from inputs to outcomes • Shift from capacity to student learning • Shift from processes to results • Concern with learning and its improvement infuses all criteria • Focus on shared responsibility • Focus on accountability, but not on reporting for its own sake • Concentration on continuous improvement

22. Higher Learning Commission Statement on Assessment of Student Learning (2003) Assessment “should be informed by the institution’s mission and include explicit public statements regarding the knowledge, skills, and competencies students should possess as a result of completing course and program requirements. . . . ”

23. What Regional Accrediting Agencies Expect:The Consortium of Regional Accrediting Commissions Report (2003) • “The institution defines quality by how well it fulfills its declared learning mission.” • The institution documents student learning by: • Setting clear learning goals • Collecting evidence of attainment • Applying collective judgment as to the meaning and utility of evidence • Using evidence to effect improvement • It uses multiple sources and kinds of evidence • It shares results with stakeholders

24. Higher Learning Commission Statement on Assessment of Student Learning (2003) Assessment’s “effectiveness in improving student learning relies on its integration into the organization’s processes for program review, departmental and organizational planning, and unit and organizational budgeting.”

25. Some Broad Themes: • Emphasis on student learning • Focus shifts from content and inputs to outcomes: what students know and value—and can do with what they know and value • General emphasis on improvement over reporting • Consideration of stakeholders • Focus on use of data and integration into institutional processes

26. Six Fundamental Questions • How are your stated student learning outcomes appropriate to your mission, programs, and degrees? • What evidence do you have that students achieve your stated learning outcomes? • In what ways do you analyze and use evidence of student learning?

27. Six Fundamental Questions • How do you ensure shared responsibility for student learning? • How do you evaluate and improve the effectiveness of your efforts to assess and improve student learning? • In what ways do you inform the public about what students learn—and how they learn it?

28. Commitment, Not Compliance • Effective assessment becomes a matter of commitment to the improvement of learning by improving the processes that produce it • Effective assessment focuses on how data can be used to improve, not on how it can be reported • Effective assessment focuses on inquiry rather than evidence • In that sense, effective assessment is at the heart of what we do as academic institutions

29. Assessment Practice Should Be: Meaningful Useful Workable Reasonable

30. Implications • Outcomes, plans, strategies, and approaches need to be focused on student learning in the context of institutional mission and values • A wide range of data-collecting strategies are possible, so long as they serve institutional needs in improving learning • Attention must be given to assessing learning in non-classroom settings—and to shared responsibility for that learning

31. Implications • Assessment strategies must be appropriate and credible, not uniform across the institution—or the state • Assessment strategies must produce information that can: --credibly be used to improve learning --adequately inform stakeholders of the degree to which outcomes have been achieved

32. Implications • Institutions must become comfortable with risk—and with the probability of failure—in the service of improved quality of learning • Assessment data must be seen as the basis for institutional conversations about learning, not as a process of self-justification • The quality, credibility, and integrity of an institution will be assumed to be reflected in its effectiveness in producing student learning

33. A Reconciling Definition: • Assessment is a transparent process in which those responsible for student learning on campus collect, analyze, and use information about student learning to improve it, while sharing information about the achievement and improvement of learning with interested stakeholders.

34. The New Assessment Is : • A process based in inquiry: • What do our students learn? • Is what they learn what we say they will learn? • How do we know they have learned? • What do we do with our knowledge? • How can we improve student learning?

35. The New Assessment Is: • Predicated on clear delineation of student learning outcomes: • What students can do with what they know • Aligned at several levels: • Institutional mission • Program/major purpose • Course’s contribution to both

36. The New Assessment Is: • A form of action research that can include: • Observations • Judgments • Validated data • Grades that correlate to outcomes

37. The New Assessment Is: • A way to improve learning • A way to improve the systems that produce learning • A way to demonstrate effectiveness to various stakeholders • A basis for conversations with stakeholders

38. The New Assessment Is: • A formalization of what many of us already do intuitively and individually • An attempt to judge achievement of essential outcomes rather than a compilation of grades or test scores • A process focused on change for improvement, not on institutional self-justification