Insights from the AHELO Feasibility Study – why, what and how

1. Insights from the AHELO Feasibility Study – why, what and how Dr Sarah Richardson Senior Research Fellow, Higher Education Australian Council for Educational Research sarah.richardson@acer.edu.au

2. Eternal challenges • How do we measure quality in teaching and learning? • How effective are institutions at educating students? • What knowledge and skills do students possess at the end of their degrees? • How do students’ learning outcomes compare to international benchmarks?

3. Final-year Bachelor degree students To determine a robust approach to measuring learning outcomes in ways that are valid across cultures and languages, and across the diversity of institutional settings and missions To give institutions data on student learning outcomes which they can use to improve quality in teaching and learning

4. To give institutions data on student learning outcomes which they can use to improve quality in teaching and learning • How do the learning outcomes of our students measure up against international benchmarks? • Benchmarking against like institutions • Students – demographics, time usage, performance, motivation, plans • Teaching – curriculum structure, class sizes, staff contact, feedback, industry experience • Institutional characteristics – size, location, funding, mission, student characteristics

5. AHELO: international participation – 17 countries (including Egypt, Kuwait and Abu Dhabi) and 12 languages (including Arabic)

6. Online 23000 students • Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? • Test fields: Economics, Engineering 250 institutions 5000 faculty

7. International agreement • Expert Groups – Engineering and Economics • Domain parameters / conceptual structure • Measurement priorities – real world • Review and map draft assessment materials • Scoring guide – desired responses • Structure reporting and review results • Generic Skills – different approach used

8. Economics learning outcomes • subject knowledge and understanding • subject knowledge and its application to real world problems • ability to make effective use of relevant data and quantitative methods • ability to communicate to specialists and non-specialists • ability to acquire independent learning skills

9. Assessment instruments • Combine pre-existing / new materials • International collaboration / resources • Adapted / created for assessment framework • Constructed response and MCQs • Pilot & qualitative feedback • Localisation – equivalency across languages • Online delivery

10. Engineering

11. Engaging participants • Countries / institutions / faculty / students • What’s in it for me? • Marketing and awareness raising • Cultural variations • Sample / population • Authority / persuasion • Incentives and inducements • Intrinsic motivation

12. Analysis and reporting • Psychometric analysis • How did the test items function? • What was the spread of difficulty? • Did difficulty vary according to student and institution characteristics? • Reporting – international, national and institutional levels

13. Using results to enhance teaching and learning • Performance by student group • Which students need greatest support? • Performance by competency • Which aspects of teaching need enhancing? • Performance by country • What is missing in the curriculum?

14. Insights from AHELO • Measure learning outcomes internationally?  • International agreement on skills and knowledge? • Assessment materials to suit diverse contexts?  • Maintain equivalence in translation? • Engage institutions, faculty and students? • Online delivery in different languages/alphabets? • Useful benchmark reports without identification? • AHELO Feasible? 

15. Selection Diagnostic Progression Outcomes Improve Quality