The “Quality Agenda:” Implications for Assessment, Policy, and Professional Accreditation

1. The “Quality Agenda:” Implications for Assessment, Policy, and Professional Accreditation Peter T. Ewell National Center for Higher Education Management Systems (NCHEMS) AACSB International Conference New Orleans, LA March 18, 2014

2. Logic of this Presentation • The “Quality Agenda:” Where Did It Come From? • The Historical Evolution of Discussions About Quality in Higher Education • An Evolving “Standards” Movement in Higher Education • National: Degree Qualifications Frameworks with the U.S. DQP as an Example • Professional: Assessment in Business Programs • Into the Future: A “Changing Ecology” for Accreditation and Its Implications for Assessing Quality

3. A Prominent “Completion Agenda” in Higher Education Policy • Stimulated by Recognition of Link Between Economic and Social Benefits and Postsecondary Attainment in Many Countries • Widespread National Attainment Goals (e.g. The Obama Goal in the U.S.: 60% of Young Adults with a Postsecondary Credential by 2020) • But Concern that “Completion” Could Come at the Expense of Academic Quality

4. Hence, a Growing “Quality Agenda” • New Interest in Aligned Student Learning Outcomes on a National and International Basis • Shift of Attention from “Doing Assessment” to “Standards for Learning” • Stakeholder Concern About Graduate Quality, Particularly from the Employment Community  So the Current Challenge is to Raise Completion Rates while Not Losing Academic Quality =

5. Changing Notions of “Quality” in Policy Discourse About Higher Education • Largely taken for granted until the emergence of formal systems of “quality assurance” like accreditation and national Quality Assurance Agencies • New conceptions of “quality” emerged gradually as colleges and universities diversified and became more complicated • Today’s conception of “quality” is thus a sedimentary construction with new notions of what should count “layered in” on top of old ones

6. First Incarnation: Reputation • Colleges for “The Quality” (who were members of a privileged elite or entering a professional class like the clergy, law, or medicine) • “College” as a reserved term: distinct from postsecondary institutions: “institutes,” “normal schools,” etc. • Alive and well on bumper stickers and big time sports teams [not to mention media rankings]

7. Second Incarnation: Resources • U.S. Accreditation Criteria of the 1920s (North Central Association): “The college should: • Enroll at least 200 students • Should comprise at least eight departments with at least one person of professorial rank • Should maintain a live, well-distributed library of at least 8000 volumes” • Quantitative resource-based criteria like these officially a thing of the past…but stuff still matters a lot in popular views of “quality”

8. Third Incarnation: Selectivity • The rise of admissions selectivity in the 1950s • Reputation and exclusivity in a new guise: in the U.S., the role of the SAT was said to be to “uncover the hidden aristocracy of talent” • An implied theory of education: smart begets smart by association and osmosis

9. Fourth Incarnation: “Fitness for Purpose” • “Mission-based” quality review emerges as most appropriate for diverse postsecondary systems • Peer review and institutional audit become the primary “assessment instruments” under this approach. • “Purposes appropriate to an institution of higher education…” • But what happened to standards?

10. Fifth Incarnation: Outcomes • An “exo-skeletal” approach: outcomes assessment largely added on to the regular processes of teaching and learning. • The resulting paradigm: • Statements of intended learning outcomes • Various ways to gather evidence of attainment • Use of the resulting information to improve • Embedded in U.S. regional accreditation standards by the mid-1990s and national QAA reviews by 2000

11. A Sixth Incarnation: Exit Proficiencies? • A common set of graduation proficiencies adopted by all providers • Assessments embedded in the regular teaching and learning process: • Signature assignments in key classes • Developed in common by teaching staff • Graded or rated using standard rubrics • Re-positioned proficiency-based transcripts that show “student learning as academic currency”

12. National Qualifications Frameworks Matrix of Identified Proficiencies by Degree Levels Purpose to Align and ‘“Moderate Academic Standards at Various Degree Levels Some Examples: Bologna Process Common Outcomes Benchmarks QFs in UK, Australia, Ireland, Scotland, and Many Others “Tuning” in Many Disciplines

13. Background to the DQP in the U.S. Qualifications Frameworks in Many Other Countries (as Noted) AAC&U LEAP Outcomes Statements and Rubrics State-Level Outcomes Frameworks in U.S. (e.g. UT, WI, CSU, ND, VA) Some Alignment of Cross-Cutting Abilities Statements Among Institutional Accreditors

14. What Does the DQP Look Like? Three Degree Levels: Associate, Bachelor’s, and Master’s Five Learning Areas: Specialized Knowledge, Broad/Integrative Knowledge, Intellectual Skills, Applied Learning, and Civic Learning Framed as Successively Inclusive Hierarchies of “Action Verbs” to Describe Outcomes at Each Degree Level Intended as a “Beta” Version, for Testing, Experimentation, and Further Development

15. An Example: Communication Skills Associate Level: The student presents substantially error-free prose in both argumentative and narrative forms to general and specialized audiences Bachelor’s Level: The student constructs sustained, coherent arguments and/or narratives and/or explications of technical issues and processes, in two media, to general and specialized audiences Master’s Level: The student creates sustained, coherent arguments or explanations and reflections on his or her work or that of collaborators (if applicable) in two or more media or languages, to both general and specialized audiences

16. Some Implications of the DQP The DQP Asserts that Every Student Should Graduate with the Designated Proficiencies. This Means that: The Typical Approach of Setting Outcomes as “Aspirations” and Conducting Assessments of “Average” Student Performance is not Enough Assessment as an “Add-On” to the Curriculum is Not Enough Assessment Must Be Embedded in Regular Student Assignments and Examination Questions and Certified at Multiple Levels on the Way to Degree Completion

17. AACSB’s “Assurance of Learning” Similar in Form and Intent to Assessment Requirements of Other Specialized Accreditors Outcomes Set by Program Consistent with Mission Both Generic and Field-Specific Outcomes Methods Selected by Program and Not Standardized Outside Consultation (e.g. with Employers) Encouraged Use of Results Emphasized as Much as Assessment Process

18. Assessment in Business: NILOA Survey Fourth of Twelve Schools or Departments with Respect to Overall Assessment Activity Fifth of Twelve in Citing “Specialized Accreditation” as Reason for Doing Assessment Second of Twelve in Using Capstone Courses Second of Twelve in Reporting High Levels of Faculty Involvement in Assessment Top School or Department Wanting Better Instruments

19. A Changing Ecology for Accreditation • New Patterns of Student Participation • New Kinds of Providers • A Transformed and Contingent Faculty • New Approaches to Instructional Provision • Constrained Resources • Most of These Forcing Learning Outcomes as the Only Practicable Way to Determine Quality

20. What are Accreditors Doing in Response? Look More Explicitly at: • Coherence of Student Experience Across Settings • How Faculty Roles are Constructed and Discharged • “Outsourcing” of Courses and Teaching • Response to Stakeholder Needs (e.g. Employers) • Transparency and Public Reporting • Use of Electronic Media and “Virtual Review”

21. Putting It All Together New Discourse About “Quality” Centered on Standards for Learning that All Students Will Achieve As Documented in an Identified Array of Proficiencies that Schools Adopt in Common Reported Publicly to a Wide Array of Stakeholders Review Techniques Transformed to Respond to New Ecology for Higher Education and New Modes of Provision