AHELO – feasibility study

1. AHELO – feasibilitystudy Informasjonsmøte i KD 1. desember 2009 Spesialrådgiver Jan S. Levy

2. Project background Why undertake the study? • After decades of quantitative growth in HE and its growing internationalisation, consensus on the need to ensure quality provision for all… but information gap on learning outcomes. Existing rankings have not focus on LO’s • Carry out a feasibility study to provide a proof of concept (Tokyo, 2008) Feasibility study goals? • To assess whether reliable cross-national comparisons of HE learning outcomes are scientifically possible and whether their implementation is feasible. • A research approach to provide a proof of concept and proof of practicality

3. Project summary: strands of work Generic skills strand Discipline strand in Engineering • Basis for discussion of expected learning outcomes delivered • through ‘Tuning’ approach. • + contextual data • International pilot test of the US Collegiate Learning Assessment (CLA), to assess the extent to which problem-solving or critical thinking can be validly measured across different cultural, linguistic and institutional contexts. • + contextual data Research-based “Value-added” or “Learning gain” measurement strand Discipline strand in Economics Several perspectives to explore the issue of value-added (conceptually, psychometrics), building on recent OECD work at school level. • Basis for discussion of expected learning outcomes delivered • through ‘Tuning’ approach. • + contextual data

4. Participation in the three strands of surveying Generic skills strand Discipline strand in Engineering Finland, Korea, Mexico, Norway • Australia, Japan, Sweden Discipline strand in Economics • Belgium (Fl.), Italy, Mexico, Netherlands

5. Project summary (cont.d) The outcomes will be used to assist countries to decide on the next steps. Why? The testing window is from August 2010 to April 2011. When? Data will be collected from a targeted population of students who are near – but before – the end of their first 3-4 year degree. Who? • Joint steering by governments, agencies and HEIs through IMHE • OECD’s role is to establish broad frameworks that guide international expert committees in charge of instrument • development. • Several options: Use existing instruments, combine existing item pools to cover frameworks, ordevelop new test material? How?

6. Feasibility study outputs A final conference and its proceedings to discuss the outcomes of the feasibility study as well as the potential impact of AHELO Less tangible: Enhanced understanding of the project and buy-in from stakeholders Three reports on the scientific and practical feasibility of implementing an international assessment focusing on discipline-specific skills in economics and engineering and on generic skills Two reports on the feasibility of reaching international consensus on defining the intended learning outcomes in two contrasted disciplines and in a cross-national fashion One report on the measurement of value-added in higher education, net of students’ incoming abilities and differences in institutional selectivity

7. Country participation - update • Critical mass attained (10 participants) • Scope for 4-5 more countries • Formal requests from 4 US states, Russian Federation • Discussions withTurkey • VolunteerrequestsfromHEIs (France, Germany, Spain) • Interest from NME non-EEA countries (Kuweit, Colombia)

8. Ledes av OECDs utdanningspolitiske komite og av IMHE GB Oppnevnt særskilt gruppe fra deltagende land (GNE) ansvarlig for gjennomføring Ekspertgrupper på de enkelte temaer OECD stiller strenge kvalitetskrav til utvikling, oversettelse og tilpasning av undersøkelses-instrumenter Konsultasjonsorgan med interessegrupper internasjonalt (EI, BIAC, TUAC, ESU, ENQA, EUA..) Undersøkelsen gjennomføres av internasjonale konsortier i samarbeid med deltagende land Internasjonal organisering

9. Scope of tendered work Initial work: Tuning-AHELO Engineering Strand (EngS) Generic Skills Strand (GenS) Economics Strand (EcoS) Value-Added Measurement Strand (VamS) Sole supplier Development of the GenSintrument (Module A) Development of the EcoSintrument (Module B) Development of the EngSintrument (Module C) • VamS • Resarch-based: • No direct assessment involved • Done through commissioned experts • (Module F) Methodological/ Instrument development Contextual dimension surveys: - student - faculty (Module D) - programme leadership - institutional leadership Initial work: Contextual Dimension Project management, survey operations and analyses of results (Module E) Field implementation and analysis

10. Two proposals received Proposal 1 Consortium led by the National Foundation for Educational Research (NFER) in partnership with Cito B.V. – National Institute for Educational Measurement, and one organisation included as subcontractor: • The Gallup Organisation Europe (Gallup Europe) Proposal 2 Consortium led by the Australian Council for Educational Research (ACER) with the following associates: • Capstan Linguistic Quality Control Agency (Belgium) • Indiana University Center for Postsecondary Research (USA) • University of Twente Centre for Higher Education Policy Studies (Netherlands) • Educational Testing Service (USA) • International Association for the Evaluation of Educational Achievement (Netherlands) • National Institute for Educational Policy Research (Japan) • Statistics Canada (Canada) • University of Florence (Italy) • Westat Inc (USA)

11. The European ClassificationofHigher Education Institutions. Welcome to U-Map. The European ClassificationofHigher Education Institutions. U-Map is thethirdphaseof a researchproject to develop a European ClassificationofHigher Education Institutions. The classificationmapsthe European highereducation landscape and will make theinstitutionaldiversityofthat landscape more transparent. U-Mapwill not rank theinstitutionsleague-table-style, butwillpositioninstitutions on a numberofdimensions, eachrepresenting an aspectofthefunction and performanceofhighereducationinstitutions. Clickhere to go to thefinder. Aboutthe Project. The conceptofdiversity has movedrapidly up thepolitical agenda over thepastdecade. The developmentofthe European Higher Education Area (EHEA) and the European Research Area (ERA) has clearlycontributed to this. The Methodology. U-Map has beendeveloped in a dynamic and interactiveprocess in which a numberof general guidelines and design principleswere used. The consultationof stakeholders was a cornerstone ofthatprocess. Watchthe demo.

12. Arbeid i Norge • Tre norske institusjoner: NTNU, UMB, HiVe • Generic skills strand • Tilbud fra CAE om å gi seminarer om CLA-undersøkelsen også til andre institusjoner • Planlagt gjennomført i februar/mars 2011 • Rapporter foreløpig planlagt til høsten 2011 • Nasjonal projektleder (NPM) – NIFU STEP i samarbeid med ILS/EKVA • Prosjektorganisert i departementet – prosjektgruppe, styringsgruppe, informasjonsmøter

13. Hvilke typer resultater for Norge? • Bidrag til den internasjonale studien • Internasjonale sammenligninger vil ikke bli laget. Utvalgene vil ikke være representative internasjonalt • Data som kan identifisere respondent blir utilgjengelige • Institusjonene må få rapport med sine resultater • Departementet må få rapport om norske institusjoner som deltok • Offentliggjøring av nasjonale resultater ikke avgjort • Ikke klart om det vil bli gitt noen form for sammenligning med internasjonal benchmark

14. TASK DESCRIPTION Performance Task Each Performance Task requires students to use an integrated set of critical thinking, analytic reasoning, problem solving, and written communication skills to answer several openended questions about a hypothetical but realistic situation. In addition to directions and questions, each Performance Task also has its own document library that includes a range of information sources, such as letters, memos, summaries of research reports, newspaper articles, maps, photographs, diagrams, tables, charts, and interview notes or transcripts. Students are instructed to use these materials in preparing their answers to the Performance Task’s questions within the allotted 90 minutes. http://www.collegiatelearningassessment.org/ http://www.cae.org/content/pdf/CLABrochure2008.pdf

15. Eksempel på performancetask