Assessing Programme Outcomes: Challenges and Possibilities

1. Workshop series to Help You Prepare Your OBA Project Proposal 17 Dec 07 2:30 pm – 5:00 pm Assessing Programme Outcomes:Challenges and Possibilities KP Kwan  2766 6320  etkpkwan@inet.polyu.edu.hk

2. Workshop outline • OBA funding: Guidelines for application • Meaning and purpose of Programme Learning Outcomes Assessment (PLOA) • Challenges and difficulties in assessing programme outcomes • Overview of possible PLOA strategies or methods to prove and improve programme effectiveness • Ideas for developing your OBA project proposal on enhancing PLOA

3. OBA funding from LTC 2007-08 • Support collaborative effort in implementing OBA • Preference for projects focusing on: • Designing effective TLA methods to facilitate students to achieve learning outcomes* • Assessing professional and generic programme outcomes* • Reviewing the alignment and mapping of subjects to programme outcomes [* Priority areas ] • Funding limit = $400K per project • Deadline: 31 Jan 2008

4. Selection criteria • Alignment with PolyU goals and priorities in implementing OBA in student learning • Outcomes/deliverables • Impact • Collaboration and engagement of front line teachers

5. Activity 1:Why are you interested? 10 minutes • In turn, briefly explain • WHY you are interested in programme learning outcomes assessment [PLOA] • WHAT you want to get out of this workshop. Sharing

6. Purpose and meaning of Programme Learning Outcomes Assessment [PLOA]

7. Why bother with PLOA? • To improve student learning • Integral component of Outcome-Based Education • Evidence and feedback for improvement • To prove or demonstrateprogramme quality and effectiveness • Professional bodies or accreditation agencies • UGC Quality Assurance Committee Audit (2010) • To showcaseprogramme and graduate quality to appeal to • Prospective and current students • Prospective employers • The community at large

8. What is PLOA? • Meaning of PLOA: • a systematic process of collecting broaderevidence or data • about how well graduates are achieving the intended programme learning outcomes, and • interpreting and using the results to improve student learning • Effectiveness/success defined in terms of outcomes actually achieved rather input or process indicators • Focus on overall programme effectiveness rather than assessing performance of individual students

9. PLOA in a nutshell • Say what we do • Do what we say • Prove it • Improve it State the PLOs that we want our graduates to achieve on completing the programme Design teaching, learning and assessment activities such that students are able to achieve the aspired PLOs Collect and present evidence to demonstrate the extent to which our graduates are actually achieving the PLOs Using outcomes evidence to improve the effectiveness of the programme in facilitating students’ achievement of the PLOs

10. Clarifying terminology • Assessment: • “… the systematic collection, review and use of information for the purpose of improving student learning.”[Marchese, 1997] • Direct or indirect • Quantitative or qualitative • Not limited to formal tests or examinations • Student learning outcomes: • “… the effect on students of the educational process – that is, the knowledge, skills, behaviours and attitudes that are imparted to students through their learning experiences.”[UGC QAC Audit Manual 2007]

11. Legitimate outcomes evidence

12. Legitimate outcomes evidence

13. Outcomes of PLOA

14. Example Intended programme learning outcome:Students should be able to identify, analyse and solve technical problems [ABET Criterion 1 Item f] • Outcome assessment methods: • Students’ self assessment of competence with respect to specific programme outcome • Specific final exam questions included in ECET207, 231, 307, 309, 417, 483 and BMET320 to measure this outcome [Target: 70% of students scored 70% or higher on each question]. [Purdue University Dept of Electrical & Computer Engineering 2007 Annual Assessment Report]

15. Example [cont’d] • Assessment findings: • 80.2% of students strongly agreed/agreed that they can perform tasks associated with this outcome. • On exam questions targeting this criterion, only 39.8% of students scored a 70% or better. • Improvement actions/changes: • The low result was traced to very poor performance in ECET284, and was addressed in the course during the next semester. • Problem solving steps will continue to be stressed, especially in foundational courses. [Purdue University Dept of Electrical & Computer Engineering 2007 Annual Assessment Report]

16. Activity 2:An audit of your PLOA activities 25 minutes • Refer to worksheet 1 • List the most important learning outcomes of your programme (PLOs) • For each of the PLOs, identify the outcomes assessment methods and evidence that are already in place that you can draw upon to evaluate the overall programme effectiveness in achieving the outcome • Reflect on your current PLOA activities and explain to your partner: • Problems you have observed • Gaps in your PLOA that you have identified Think-pair-share

17. Challenges and difficulties

18. Problem of measurement “ Not everything that can be counted counts, and not everything that counts can be counted.” Albert Einstein

19. Limitation of subject assessments • Subject grades alone do NOT provide adequate information for PLOA: • Graded on students’ performance w.r.t. subject-specific content and outcomes • Seldom assess broader programme outcomes that cut across subjects or require integration of learning from different subjects • Generic competencies not assessed and reported separately, if assessed at all • Reflected short-term learning rather long-term learning outcomes or impact of programme • Some outcomes are difficult to be measured via conventional teacher-marked subject assessments

20. Benchmarking and standard • Criteria for success:What explicit criteria and standard would you use to indicate programme success in reaching the intended PLOs? • Benchmarking:How to demonstrate/prove that the level of achievement and competencies of your graduates regarding the PLOs are at par with international standard and what is expected for a beginning professional in your field?

21. Other problems/difficulties • Accountability-driven • Too many outcomes • Lack of systematic outcomes evidence • Mostly indirect survey data • Data not available or presented in ways for those who need to analyse and apply it • Exit outcomes measures rather than value-added outcomes measures • Lack of involvement of front-line staff • Workload, time and resources for outcomes assessment Just another routine bureaucratic chore !

22. Best practices in PLOA • Improvement-oriented • Owned, driven and shared by frontline academic staff • Must include direct measures of students’ performance • Use both direct and indirect evidence from multiple sources • Involve stakeholders in benchmarking standard and evaluating programme outcomes • Focus more on high-priority outcomes • Not doingMORE, but doing it DIFFERENTLY and EFFECTIVELY

23. Developing a PLOA Plan

24. 6-step approach to developing a PLOA • Define programme mission and objectives • Articulate the intended programme learning outcomes • Choose appropriate outcomes assessment measures and methods • Explain how the outcomes evidence will be collected, analysed and reported [what, when, how and by whom] • Describe how the results will be used to improve learning • Establish timeline for implementation

25. Possible PLOA strategies forcollecting directevidence of learning outcomes • Course-embedded assessment • Capstone experience/project • Clinical/professional performance appraisals • Portfolio assessment • Standardised tests or exams (e.g. Major Field Achievement Tests, GRE, IELTS, CLA, CCTST, ATTA, etc.) • Licensure exams(e.g. HKICPA Exam) • External examiner or expert panel review(of student work samples)

26. Subject grading vs PLOA

27. Subject grading vs PLOA

28. Subject grading vs PLOA

29. Subject grading vs PLOA

30. Assessing for grading and PLOA Subject grade for each student [for students’ acad achievement record] Subjectteacher Grading of students’ work for subject assessment

31. Assessing for grading and PLOA Subject grade for each student [for students’ acad achievement record] Subjectteacher Grading of students’ work for subject assessment Evidence of aggregate performance of students on particular PLOs [for Dept or Programme review] Subjectteacher Re-assessing students’ work for outcomes assessment ORDept review panel, student peers, employers…

32. Possible PLOA strategies forcollecting indirectevidence of learning outcomes • Students’ self-assessment of learning gains(e.g. the PolyU SAARD administered by SAO) • Surveys and interviewsof employers, alumni/graduates, current students • Students’ engagement in curricular, co-curricular and extra-curricular activities (e.g. community service learning) • Institutional data(e.g. completion/progression rate, employment statistics, etc.) • Standardised surveys • National Survey of Student Engagement • College Student Experience Survey • Course Experience Questionnaire

33. Example of acomprehensive PLOA plan

34. Activity 3:Planning for enhancing PLOA 20 minutes • Refer to the PLOA problems or gaps you identified in Activity 2 • What new or changed PLOA measures or processes will be most appropriate for addressing the problems or gaps? • Explain briefly your project ideas to your partner: • What is the problem/gap? • What outcomes assessment activity will be added/modified? How? • Who will be involved? • How will you evaluate its usefulness or impact? Think-pair-share

35. Developing Your OBA Project Proposal

36. Good PLOA project proposals • Designed to address important gaps/problems in current PLOA measure or practice • Based/modelled on international ‘best practices’ • System-wide and sustainable impact • Participatory and collaborative (within Programme, Department or Faculty/School) • Involvement of key stakeholders • Systematic project evaluation plan • Effective plan to disseminate deliverables/experience

37. Ideas for potential projects • Develop a comprehensive PLOA plan • Design and pilot innovative PLOA activities or measures, e.g. • Course-embedded assessment • Capstone course/projects • Portfolio assessment, etc. • Refine/Design processes to collect and use outcomes evidence for improving programme outcomes • e.g. assessment committee, PLOA report, outcomes-based programme review • Benchmarking graduates’ competencies • Involving employers and professionals in benchmarking standards and assessing graduates in key PLOs

38. Other ideas • Other ideas and suggestions? • Further questions?

39. Consultation and support • KP Kwan EDCx6320 TU612etkpkwan@inet.polyu.edu.hk

40. Useful references • Programme-Based Review and Assessment: Tools and Techniques for Program Improvement [University of Massachusetts Amherst] Available at:http://www.umass.edu/oapa/oapa/publications/online_handbooks/program_based.pdf • California Assessment Institute Resourceshttp://cai.cc.ca.us/Resources/index.htm • Allen, M. (2004). Assessing Academic Programs in Higher Education. Anker Publishing Co. Ltd. • Walvoord, B.E. (2004). Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education. Jossey-Bass.