Program Assessment Workshop

1. Program Assessment Workshop Mary Allen Qatar University September 2011

2. Learning Outcomes for this Workshop Workshop participants will be able to: • draft/revise learning outcomes • develop/analyze curriculum maps • develop/refine sustainable, multi-year assessment plans • develop/refine rubrics and calibrate reviewers • analyze assessment results • use a variety of strategies to close the loop • evaluate the impact of improvement actions

3. Assessment • an on-going process designed to monitor and improve student learning

4. Assessment Faculty: • Develop SLOs • Verify curriculum alignment • Develop an assessment plan • Collect evidence • Assess evidence and reach a conclusion • Close the loop

5. SACS Expectation for Assessment • Standard 3.3.1

6. Learning-Centered Institutions • Program goals • Cohesive curriculum • How students learn • Course structure and pedagogy • Faculty instructional role • Assessment • Campus support for learning

7. Quotations from the Wise and Experienced

8. Assessment Vocabulary • Direct vs. indirect assessment • Value-added vs. absolute learning outcomes • Authentic assessment • Formative vs. summative assessment • Triangulation

9. If you have absolute outcomes, your assessment plan should emphasize direct, authentic, summative assessment, with triangulation.

10. Learning Outcomes • Clarify what faculty want students to learn • Clarify how each outcome can be assessed

11. Learning Outcomes • Knowledge • Skills • Attitudes/Values/Predispositions

12. Outcome Levels • CLO • PLO • ILO

13. Possible Learning Goals

14. Examples of Outcomes

15. Goals and Outcomes:Outcomes and Performance Indicators • List of goals and outcomes • List of outcomes • Typically 6-8 outcomes in all

16. Bloom’s Taxonomy

17. Creating Quality Outcomes • Active verbs • Simple language • Real vs. aspirational • Aligned with mission • Avoid compound outcomes • Outcomes vs. learning processes • Focus on high-priority learning

18. The Cohesive Curriculum • Coherence • Synthesizing experiences • On-going practice of learned skills • Opportunities to develop increasing sophistication and to apply what is learned

19. Curriculum Map • I = Introduced • D = Developed & Practiced with Feedback • M = Demonstrated at the Mastery Level Appropriate for Graduation

20. Curriculum Map Patterns • Curriculum Map 2 • Curriculum Map 3

21. Entries on the Map Indicate: • CLOs that align with relevant PLOs • Faculty can provide artifacts for assessment • Faculty teach courses consistent with the map

22. The Curriculum Map: • Focuses faculty on curriculum cohesion • Guides course planning • Allows faculty to identify potential sources of assessment evidence • Allows faculty to identify where they might close the loop

23. We don’t have to assess every outcome in every student every year! • Except for NCATE-accredited programs

24. Assessment Plan • Who? • What? • Where? • When? • Why?

25. Sampling • Relevant samples • Representative samples • Reasonably-sized samples

26. Ethical Issues • Anonymity • Confidentiality • Privacy • Informed consent

27. Sample Assessment Plan Find examples of: • Direct assessment • Indirect assessment • Formative assessment • Summative assessment • Authentic assessment • Triangulation

28. Assessment Plan Template • PLO • When to assess • What direct and indirect evidence to collect • Who will collect the evidence • How evidence will be assessed • How decisions will be made

29. Properties of Good Assessment • Valid • Reliable • Actionable • Efficient and cost-effective • Engages students • Interesting to faculty • Triangulation

30. Direct Assessment Strategies • Published tests • Locally-developed tests • Embedded assessment • Portfolios

31. Indirect Assessment Strategies • Surveys • Interviews • Focus groups

32. Rubrics • Holistic • Analytic

33. Rubric Examples • Rubric Packet • AAC&U VALUE Rubrics • Specialized Packets

34. Rubric Strengths • Efficiency • Defines faculty expectations • Well-trained reviewers use the same criteria • Criterion-referenced judgments • Ratings can be done by multiple people

35. Two Common Ways to Apply Rubrics • Assess while grading • Assess in a group

36. Assessing and Grading Simultaneously • Columns are used for assessment • Faculty can adapt an assessment rubric in different ways • Faculty maintain control over their own grading

37. Turn to someone near you and explain how you can grade and assess simultaneously.

38. Assessment vs. Grading • Grading may require extra criteria • Grading requires more precision • Calibrate when doing assessment

39. Rubrics Can: • Speed up grading • Clarify expectations to students • Reduce student grade complaints • Improve the reliability and validity of assessments and grades • Make grading and assessment more efficient and effective • Help faculty create better assignments

40. Suggestions for Using Rubrics in Courses

41. Typical Four-Point Rubric Levels • Below Expectations • Needs Improvement • Meets Expectations • Exceeds Expectations

42. Rubric Category Labels

43. Creating a Rubric • Adapt an already-existing rubric • Analytic method

44. Drafting the Rubric • Consider starting at the extremes • Some words I find useful

45. Managing Group Readings • One reader/document. • Two independent readers/document. • Paired readers.

46. Before Inviting Colleagues: • Collect the assessment evidence and remove identifying information. • Develop and pilot test the rubric. • Select exemplars of weak, medium, and strong student work. • Consider pre-programming a spreadsheet so data can be entered and analyzed during the reading and participants can discuss results immediately.

47. Inter-Rater Reliability • Correlation • Discrepancy Index

48. Rubric Orientation and Calibration

49. Assessment Standards • How good is good enough?

50. Closing the Loop • Celebrate! • Change pedagogy • Change curriculum • Change student support • Change faculty support • Change equipment/supplies/space