2012 Asia -Pacific Quality Network Conference Siem Reap Angkor, 29 February – 2 March 2012

1. OECD feasibilitystudy for an international Assessment of HigherEducation Learning Outcomes (AHELO) 2012 Asia-Pacific Quality Network Conference SiemReap Angkor, 29 February – 2 March 2012 Karine Tremblay, OECD

2. AHELO Rationale Feasibilitystudyoverview Emerginginsights from Phase 1 Lookingahead

3. Higher education today… Mass access Internationalisation of HE and high-skilled labour markets Concern for drop out (3 out of 10 studentsacross the OECD) Equity remains an issue

4. AHELO rationale • Despite huge progress in quality assurance, institutional quality remains largely unknown • No perfect proxy • Reputation race • Rankings biased • Satisfaction culturally sensitive • Labour market outcomes problematic Shared vision of OECD Education Ministers (2006) Shift from quantity to quality Quality information needed for all sorts of reasons So what? • Information vacuum filled by available information • Learning outcomes need to be taken into account • Defining them (Tuning) • Incorporating them in quality assurance • Measuring them (AHELO)

5. Why a feasibility study? Assess whether we can measure what undergrad. students know and can do upon graduation … … across diverse countries, languages, cultures and types of institutions Test the practicality of implementation Test the science of the assessment

6. AHELO Rationale Feasibilitystudyoverview Emerginginsights from Phase 1 Lookingahead

7. AHELO: 4 strands of work Generic skills strand Discipline strand in Engineering Exploring the feasibility of measuringlearningoutcomesin 2 contrasted disciplines to prove concept Research-based “Value-added” or “Learning gain” measurement strand Discipline strand in Economics Several perspectives to explore the issue of value-added (conceptually, psychometrics), building on similar work at school level. Critical to strive in 21st Century knowledgesocieties

8. A range of geographic, linguistic and cultural backgrounds involved • Generic skills • Economics • Engineering Observers: Bahrein Brazil SaudiArabia Singapore

9. Work to be undertaken in 2 phases Generic Skills Framework Economics Framework Engineering Framework Frameworks Phase 1 -Initial proof of concept Instrument development & small-scale validation Generic Skills Instrument Economics Instrument Engineering Instrument Jan 2010- June 2011 Contextual dimension surveys Phase 2 -Scientific feasibility & proof of practicality Project management, survey operations and analyses of results Implementation Mar 2011- Dec 2012 Where we are now

10. Where do we stand now?

11. AHELO Rationale Feasibilitystudyoverview Emerginginsights from Phase 1 Lookingahead

12. Feasibility of instrumentation (criteria) Generic skills OK • Framework development • Instrument development and mapping to framework ONGOING Economics and Engineering OK • Framework development • Instrument development and mapping to framework OK Contextual dimension OK • Framework development • Instrument development and mapping to framework OK

13. Feasibility of implementation (criteria) Overall ONGOING • Valid and feasibleassessment design • Practicallyfeasiblestudy • Methodologicallyrigorousimplementation • Engagement of faculties and students • Successful instrument delivery • Successfulsurveyoperations in countries and HEIs • Successful scorer training for comparability • Institution reports valuable for participatingHEIs and countries ONGOING ONGOING ONGOING ONGOING ONGOING ONGOING ONGOING

14. AHELO Rationale Feasibilitystudyoverview Emerginginsights from Phase 1 Lookingahead

15. Beyond the feasibility study Assuming positive outcomes of the feasibility study… • Qualitative proof of concept • Scientific feasibility (quantitative/psychometric focus) • Feasibility of implementation No preset outcome, but reflection on AHELO longer term development motivated by… • Encouraging interim assessment to date • Growing interest in the work worldwide, not only from Governments but from institutions and other stakeholders

16. Imagine 2017… … Learning outcomes data feeding into external QA evaluations • Policy recommendations on Quality Assurance from the OECD Thematic Review of Tertiary Education (2008): • Improve quality information base • Increase focus on student outcomes … From diagnosis to treatment/improvement at institutional level • Better information on student learning outcomes is the first step to • improve teachingand learning for all: • Provide evidence for national and institutional policy and practice • Equip institutions with the method and tools to improve teaching

17. Enhance quality assurance … Importance of contextual data and subsequent analyses • AHELO to go beyond snapshot diagnosis • To enable analysis of what is distinctive of high-performing institutions and identification of best practice • Contextsurvey questionnaires fromstudents, faculties and institution leaders • Towardsidentifyingwhatworks, for whichstudents and in whichcontexts. Hugepotential in terms of equity and responsiveness

18. Foster transparency and mobility … Strengthen attractiveness and competitiveness of HE system • Policy recommendations on internationalisation from the OECD Thematic Review of Tertiary Education (2008): • Enhance the international comparability of higher education • Develop alternatives to current global rankings • Improve information to prospective international students … Importance of international LO assessments for credit transfer, degree recognition and mobility Towards better portability of HE credentials.

19. Current sponsors

20. For more information, visit www.oecd.org/edu/ahelo Karine.Tremblay@oecd.org