400 likes | 410 Views
Join this workshop to learn about assessing programme outcomes and challenges in order to enhance your OBA project proposal. Topics include PLOA strategies, effective TLA methods, and aligning subjects with programme outcomes.
E N D
Workshop series to Help You Prepare Your OBA Project Proposal 17 Dec 07 2:30 pm – 5:00 pm Assessing Programme Outcomes:Challenges and Possibilities KP Kwan 2766 6320 etkpkwan@inet.polyu.edu.hk
Workshop outline • OBA funding: Guidelines for application • Meaning and purpose of Programme Learning Outcomes Assessment (PLOA) • Challenges and difficulties in assessing programme outcomes • Overview of possible PLOA strategies or methods to prove and improve programme effectiveness • Ideas for developing your OBA project proposal on enhancing PLOA
OBA funding from LTC 2007-08 • Support collaborative effort in implementing OBA • Preference for projects focusing on: • Designing effective TLA methods to facilitate students to achieve learning outcomes* • Assessing professional and generic programme outcomes* • Reviewing the alignment and mapping of subjects to programme outcomes [* Priority areas ] • Funding limit = $400K per project • Deadline: 31 Jan 2008
Selection criteria • Alignment with PolyU goals and priorities in implementing OBA in student learning • Outcomes/deliverables • Impact • Collaboration and engagement of front line teachers
Activity 1:Why are you interested? 10 minutes • In turn, briefly explain • WHY you are interested in programme learning outcomes assessment [PLOA] • WHAT you want to get out of this workshop. Sharing
Purpose and meaning of Programme Learning Outcomes Assessment [PLOA]
Why bother with PLOA? • To improve student learning • Integral component of Outcome-Based Education • Evidence and feedback for improvement • To prove or demonstrateprogramme quality and effectiveness • Professional bodies or accreditation agencies • UGC Quality Assurance Committee Audit (2010) • To showcaseprogramme and graduate quality to appeal to • Prospective and current students • Prospective employers • The community at large
What is PLOA? • Meaning of PLOA: • a systematic process of collecting broaderevidence or data • about how well graduates are achieving the intended programme learning outcomes, and • interpreting and using the results to improve student learning • Effectiveness/success defined in terms of outcomes actually achieved rather input or process indicators • Focus on overall programme effectiveness rather than assessing performance of individual students
PLOA in a nutshell • Say what we do • Do what we say • Prove it • Improve it State the PLOs that we want our graduates to achieve on completing the programme Design teaching, learning and assessment activities such that students are able to achieve the aspired PLOs Collect and present evidence to demonstrate the extent to which our graduates are actually achieving the PLOs Using outcomes evidence to improve the effectiveness of the programme in facilitating students’ achievement of the PLOs
Clarifying terminology • Assessment: • “… the systematic collection, review and use of information for the purpose of improving student learning.”[Marchese, 1997] • Direct or indirect • Quantitative or qualitative • Not limited to formal tests or examinations • Student learning outcomes: • “… the effect on students of the educational process – that is, the knowledge, skills, behaviours and attitudes that are imparted to students through their learning experiences.”[UGC QAC Audit Manual 2007]
Example Intended programme learning outcome:Students should be able to identify, analyse and solve technical problems [ABET Criterion 1 Item f] • Outcome assessment methods: • Students’ self assessment of competence with respect to specific programme outcome • Specific final exam questions included in ECET207, 231, 307, 309, 417, 483 and BMET320 to measure this outcome [Target: 70% of students scored 70% or higher on each question]. [Purdue University Dept of Electrical & Computer Engineering 2007 Annual Assessment Report]
Example [cont’d] • Assessment findings: • 80.2% of students strongly agreed/agreed that they can perform tasks associated with this outcome. • On exam questions targeting this criterion, only 39.8% of students scored a 70% or better. • Improvement actions/changes: • The low result was traced to very poor performance in ECET284, and was addressed in the course during the next semester. • Problem solving steps will continue to be stressed, especially in foundational courses. [Purdue University Dept of Electrical & Computer Engineering 2007 Annual Assessment Report]
Activity 2:An audit of your PLOA activities 25 minutes • Refer to worksheet 1 • List the most important learning outcomes of your programme (PLOs) • For each of the PLOs, identify the outcomes assessment methods and evidence that are already in place that you can draw upon to evaluate the overall programme effectiveness in achieving the outcome • Reflect on your current PLOA activities and explain to your partner: • Problems you have observed • Gaps in your PLOA that you have identified Think-pair-share
Problem of measurement “ Not everything that can be counted counts, and not everything that counts can be counted.” Albert Einstein
Limitation of subject assessments • Subject grades alone do NOT provide adequate information for PLOA: • Graded on students’ performance w.r.t. subject-specific content and outcomes • Seldom assess broader programme outcomes that cut across subjects or require integration of learning from different subjects • Generic competencies not assessed and reported separately, if assessed at all • Reflected short-term learning rather long-term learning outcomes or impact of programme • Some outcomes are difficult to be measured via conventional teacher-marked subject assessments
Benchmarking and standard • Criteria for success:What explicit criteria and standard would you use to indicate programme success in reaching the intended PLOs? • Benchmarking:How to demonstrate/prove that the level of achievement and competencies of your graduates regarding the PLOs are at par with international standard and what is expected for a beginning professional in your field?
Other problems/difficulties • Accountability-driven • Too many outcomes • Lack of systematic outcomes evidence • Mostly indirect survey data • Data not available or presented in ways for those who need to analyse and apply it • Exit outcomes measures rather than value-added outcomes measures • Lack of involvement of front-line staff • Workload, time and resources for outcomes assessment Just another routine bureaucratic chore !
Best practices in PLOA • Improvement-oriented • Owned, driven and shared by frontline academic staff • Must include direct measures of students’ performance • Use both direct and indirect evidence from multiple sources • Involve stakeholders in benchmarking standard and evaluating programme outcomes • Focus more on high-priority outcomes • Not doingMORE, but doing it DIFFERENTLY and EFFECTIVELY
6-step approach to developing a PLOA • Define programme mission and objectives • Articulate the intended programme learning outcomes • Choose appropriate outcomes assessment measures and methods • Explain how the outcomes evidence will be collected, analysed and reported [what, when, how and by whom] • Describe how the results will be used to improve learning • Establish timeline for implementation
Possible PLOA strategies forcollecting directevidence of learning outcomes • Course-embedded assessment • Capstone experience/project • Clinical/professional performance appraisals • Portfolio assessment • Standardised tests or exams (e.g. Major Field Achievement Tests, GRE, IELTS, CLA, CCTST, ATTA, etc.) • Licensure exams(e.g. HKICPA Exam) • External examiner or expert panel review(of student work samples)
Assessing for grading and PLOA Subject grade for each student [for students’ acad achievement record] Subjectteacher Grading of students’ work for subject assessment
Assessing for grading and PLOA Subject grade for each student [for students’ acad achievement record] Subjectteacher Grading of students’ work for subject assessment Evidence of aggregate performance of students on particular PLOs [for Dept or Programme review] Subjectteacher Re-assessing students’ work for outcomes assessment ORDept review panel, student peers, employers…
Possible PLOA strategies forcollecting indirectevidence of learning outcomes • Students’ self-assessment of learning gains(e.g. the PolyU SAARD administered by SAO) • Surveys and interviewsof employers, alumni/graduates, current students • Students’ engagement in curricular, co-curricular and extra-curricular activities (e.g. community service learning) • Institutional data(e.g. completion/progression rate, employment statistics, etc.) • Standardised surveys • National Survey of Student Engagement • College Student Experience Survey • Course Experience Questionnaire
Activity 3:Planning for enhancing PLOA 20 minutes • Refer to the PLOA problems or gaps you identified in Activity 2 • What new or changed PLOA measures or processes will be most appropriate for addressing the problems or gaps? • Explain briefly your project ideas to your partner: • What is the problem/gap? • What outcomes assessment activity will be added/modified? How? • Who will be involved? • How will you evaluate its usefulness or impact? Think-pair-share
Good PLOA project proposals • Designed to address important gaps/problems in current PLOA measure or practice • Based/modelled on international ‘best practices’ • System-wide and sustainable impact • Participatory and collaborative (within Programme, Department or Faculty/School) • Involvement of key stakeholders • Systematic project evaluation plan • Effective plan to disseminate deliverables/experience
Ideas for potential projects • Develop a comprehensive PLOA plan • Design and pilot innovative PLOA activities or measures, e.g. • Course-embedded assessment • Capstone course/projects • Portfolio assessment, etc. • Refine/Design processes to collect and use outcomes evidence for improving programme outcomes • e.g. assessment committee, PLOA report, outcomes-based programme review • Benchmarking graduates’ competencies • Involving employers and professionals in benchmarking standards and assessing graduates in key PLOs
Other ideas • Other ideas and suggestions? • Further questions?
Consultation and support • KP Kwan EDCx6320 TU612etkpkwan@inet.polyu.edu.hk
Useful references • Programme-Based Review and Assessment: Tools and Techniques for Program Improvement [University of Massachusetts Amherst] Available at:http://www.umass.edu/oapa/oapa/publications/online_handbooks/program_based.pdf • California Assessment Institute Resourceshttp://cai.cc.ca.us/Resources/index.htm • Allen, M. (2004). Assessing Academic Programs in Higher Education. Anker Publishing Co. Ltd. • Walvoord, B.E. (2004). Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education. Jossey-Bass.