Assessment that Matters: Creating Authentic Academic Cultures of Evidence

1. Assessment that Matters: Creating Authentic Academic Cultures of Evidence Peter T. Ewell National Center for Higher Education Management Systems

2. Original Roots of �Assessment� Re-Capturing Coherence in the Curriculum From Core Curriculum to Coursework �Bazaar� Faculty Isolation in the Classroom Growing Demands for �Learning Productivity� Public Accountability for �Results� TQM/CQI Parallels in Business and Industry Resulting Tensions in Both Motive and Method Ever Since�.

3. Why Not Grades? Can�t Communicate About the Outcomes of Groups of Courses Lack of Agreed-Upon Content/Outcome Referents [�Standards I�] Judgements are Inter-subjectively Unreliable [�Standards II�]

4. Three Enduring Characteristics of Assessment Assessment is About Learning Focus on Cognitive Abilities Behaviors Useful, but Secondary Assessment is About Performance Focus on Concrete Abilities--What Students Can Do Requires Clear Definitions of Performance [Appropriate to Context in Which Ability Will be Used] Assessment Goes Beyond Individual Students Focus on Curriculum and Group Performance Requires More than Summative Judgements

5. More Recent Influences on Assessment Growing Knowledge About How Teaching and Learning Really Works Cognitive Science Instructional �Good Practice� New Approaches to Instruction Technology and New Media �Learner-Centric� Approaches Increased Emphasis on Quality Assurance K-12 Standards Movement Competency-Based Certification

6. Assessment Approaches: A Resulting Shift in Emphasis Accountability-Based: Assessments Added onto Instruction to �Check Up� on the System in the Aggregate Scholarship and Continuous Improvement: Assessments Built Into the System to Simultaneously Assure Standards and Provide Feedback on Collective Performance

7. An Expanded Vision of Assessment �Seamless� Assessments are an Integral Part of Curriculum and Pedagogy Assessments Emphasize Connections and Longitudinal Development, not Just Attainment Assessments Look at Processes and Experiences as well as Outcomes

8. Examples of Assessment Technique Consistent with this View Capstone Courses and Experiences Student Portfolios [Increasingly Electronic] Course-Embedded Assessments and Dimensional Grading of Selected Class Assignments Longitudinal Studies and Analyses of the Effectiveness of Pre-Requisite Sequences

9. Commonly-Encountered Issues Creating Appropriate Assignments Judging Student Work Consistently and Usefully Keeping the Process Manageable Creating and Maintaining a �Culture of Evidence�

10. Using Class Assignments: A Basic Distinction Direct Assessment: Assignment is Constructed to Explicitly Address the Ability in Question Indirect Assessment: Work Samples are Selected that [Hopefully] Reveal the Ability in Question

11. Types of Assignments Suitable as �Vehicles� for Assessment Common Course Examinations Research/Topical Summaries Short Papers Presenting or Critiquing an Argument or Position Journals or Reflective Commentaries Individual or Group Oral Presentations Portfolios

12. Looking at Student Work: Some Basic Principles Ensure that the Assignment Actually Elicits the Ability in Question Think About Separating �Official� Grades from Assessment Evidence Incorporate Multiple Judgement Incorporate Multiple Dimensions of the Ability in Question

13. Looking at Student Work: Some Tools Ability Checklists Holistic Scoring Rubrics Dimensional Scoring Rubrics

14. Looking at Student Work: Some Operational Guidelines Start with Real Student Work Rate Samples Using an Overall Binary Judgement (e.g. �Adequate/Inadequate�) Work Outward Toward Additional Dimensions of Performance Work Outward Toward Additional Levels of Performance Anchor Each Level with Actual Examples of Student Work

15. Some Guidelines for Keeping Assessment Manageable Inventory What You are Already Doing Utilize Existing Points of Contact with Students Select and Sample Drop What You Don�t Use

16. Some Prominent Pitfalls Linear Thinking Goals Don�t Always Come First Application and Action Needed at All Stages The �Perfect Data Fallacy� Recognize that All Data are �Bad� How Good is Good Enough to Inform Action? Fixed Targets and Standards Look at Strengths and Weaknesses Disaggregate Emphasize Continuous Improvement Either Excessive or Non-Existent Consequences The �Paranoia Shift� Emphasize Collective Benefits

17. Cultures of Evidence: Success Factors Visible Metaphor of Scholarship Emphasizing �Evidence� Over �Measurement� Beginning with Real Problems and Processes, not with �Method� Consistent Messages [and Incentives] from Leadership Periodically �Re-Socializing� the Community �Closing the Loop� with Action Turn now to what we think we have learned about some of the practicalities of making assessment processes work��tactics� of assessment in any setting.. - Begin with factors inducing and inhibiting success.. - Then move to the particularly delicate topic of what the relationship ought to be between quality assurance agencies and institutions (or internally between assessment offices and units/faculties) in order to get real engagement and improvement.. [SLIDE] Turn now to what we think we have learned about some of the practicalities of making assessment processes work��tactics� of assessment in any setting.. - Begin with factors inducing and inhibiting success.. - Then move to the particularly delicate topic of what the relationship ought to be between quality assurance agencies and institutions (or internally between assessment offices and units/faculties) in order to get real engagement and improvement.. [SLIDE]

18. Some Final Reminders About Assessment:Assessment is�.. Not �Measuring Everything that Moves� Not Just �Checking up After the Fact� Not Searching for �Final� Answers Not Always Being as �Precise� as Possible Not Ever Expecting to be Done

20. Four Design Principles for Constructing Program-Level Assessments Comprehensiveness: Assessment procedures should cover knowledge and skills taught throughout the program Multiple Judgments: Assessment procedures should involve more than one source of evidence or involve multiple judgments of student performance Multiple Dimensions: Assessment procedures should provide information on multiple dimensions of student performance, i.e., they should yield more than a single summative grade Directness of Assessment: Assessment procedures should involve at least one type of evidence based on direct observation or demonstration of student capacities; i.e., they should involve more than simply a self-report