1 / 14

Study at a glance

Assessing higher education learning outcomes globally Professor Hamish Coates hamishc@unimelb.edu.au. Are valid and reliable comparisons of learning outcomes scientifically and practically feasible?. Study at a glance. An international feasibility study – not a ‘pilot’.

marlon
Download Presentation

Study at a glance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessing higher education learning outcomes globallyProfessor Hamish Coateshamishc@unimelb.edu.au

  2. Are valid and reliable comparisons of learning outcomes scientifically and practically feasible? Study at a glance • An international feasibility study – not a ‘pilot’ • Conceived ~2005, announced 06, scoping 06-09, work 10-13 • Testing 23000 ‘bachelor degree’ students, with data from 5000 faculty, 25 national managers, 250 institutional coordinators, 17 policymakers, and hundreds of advisors and stakeholders • Designed, developed, validated and evaluated frameworks, tests, infrastructure and processes, etc. • HEI reports (not for systems), along with international reports and materials, and recommendations for a main study

  3. Linguistically diverse… • Arabic Dutch English Finish • Flemish Italian Japanese Korean • Norwegian Russian Slovak Spanish

  4. Instrument architecture Generic Skills Economics Engineering Contextual Dimension National • Is it feasible to develop frameworks and instruments to test discipline-specific learning outcomes? • Test fields: Economics, Engineering Institution Faculty Student

  5. Framework and item development Tuning AHELO / QAA frameworks Curriculum documents Accreditation systems Discipline research • Item creation • Gather existing materials • Item workshops • Technical review • Framework mapping • Adaptation, translation • Verification • Outcome specification • Document analysis • Consultation • Synthesis, review • Qualitative testing • Quantitative testing • Operationalisation • Instrument validation • Framework creation Authentic, hybrid item types ‘Above content’ reasoning

  6. Translation, adaptation and verification Source version Translation 1 Translation 2 Designed to maintain cross-national comparability of assessment materials A holistic, robust and flexible approach, linked with item production and validation Reconciliation Economists or engineers who are speakers of target language Verification: verifier (linguist) and domain specialist Adaptations managed as a continuous process National review Native speakers of target language trained to detect specific pitfalls Final check

  7. Psychometric reviews

  8. Student feedback for a task

  9. Leadership: International project management and supporting national teams • Operationalisation: Preparing tests and context instruments for secure online delivery, and training makers • Sampling: Engaging institutions, sampling faculty and students, and quality assurance Project implementation • Assessment: Supporting national training, managing testing, and managing marking quality • Reporting: Compiling data products, psychometric and statistical analysis, and system, institution and stakeholder reports • Evaluating: Scientific and practical feasibility, recommendations for full-scale study

  10. Country communications

  11. Defined engagement cycle established to support systems, institutions and students… but not faculty • Broad insights • Designed sustainable business models for such work… but other models • Assessment frameworks and test instruments developed to support multidimensional test/context instrumentation • Established methods for test design, development, translation/adaptation and validation • Defined operational workflow and quality control procedures required to support global testing • Forming awareness of how such work is positioned globally

  12. Institutional positioning Change horizons • System monitoring • Faculty improvement • AHELO • Stakeholder engagement • Policy research • Learner feedback

  13. Assessing higher education learning outcomes globallyProfessor Hamish Coateshamishc@unimelb.edu.au

More Related