Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models

Best practices in continuous evaluation:Evidence-Based Decision Making and Admissions Models Admissions criteria and success criteria ETS Europe Jenny Dalalakis jdalalakis@etseurope.org January 2006 Gerben van Lent gvanlent@etseurope.org

Key Messages • Admissions Criteria are the outcome of a decision process which must be regularly evaluated • The process has the following components: • Strategic objectives of the institution • Instruments to identify student quality • Student application model • Admissions criteria

How to Admit? What is Student Quality? How to Decide? Objective of this presentation: Introduce Best Practices in Admission Evaluations Why Admit?

C. How to Admit? Admissions Criteria A. What is Student Quality? Qualities sought in student body B. How to Decide? Evaluation Models, Fairness, Validity Admissions: Best Practices Why Admit? Consensus on Strategic Objectives

Outline Determining Institute’s Strategic Objectives for Admissions Deciding on Qualities of Student Body Evaluating Application Models Deciding on Key Admissions Criteria Wrap up: Conclusions and Questions

Strategic Objective: Successful fit between Policy and Admissions Process College Board research identified best practices as: • Consensus on admissions objectives • Explicitly affirmed and revised periodically • Quantitative and qualitative criteria of incoming student body • Consensus on the qualities sought in student body • Board, staff, alumni, current student body, labour market representatives • Academic and non-academic qualities of admired students • Acceptable instruments to measure these desired qualities • Admissions criteria • Institutional objectives supported by individual and group qualities of student body • Reflected in application components (explicit) and underlying factors (role of informal priorities) • Evaluations and processing models • Institutional goals supported in transparent manner • Institution’s Policy affects weight given to student quality measurements • Profile of student balanced among different qualities sought • Fairness • relevance of information, equal access, transparency of admission process • Validity of admissions process • Periodic correlation measurement • Future considerations • change is inevitable Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)

Strategic Objectives: Why Admit? ACA Research identified 6 key objectives • Policy • Revenue • Profile • Sustainability • Non-discrimination • Demand (Hybrid needs typically exist) Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)

Why Admit? Key objective: Development Policy Institution Policy or Government Policy requires foreign students • France: Prestige in educating other countries’ elite students and future leaders • Netherlands: Manner of providing aid to developing countries • UK, US: Foreign students become host country’s goodwill ambassadors when they return home • EU: Support for mobility (Bologna, Lisbon) means international student exchanges are vital at Higher Education level Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)

Why Admit? Key objective: Revenue Market-driven need for funds related to foreign students • Public and private schools may charge more for all or some international students directly so the extra income is welcome • UK, US • CH (MBA programme students specifically) • NL (English-taught programme students specifically) • Public university with foreign students may qualify for extra government funds • Australia • Programme survival in danger and loss of staff so foreign students help numbers • SE, CH, NL • Disincentives may apply if institution does not qualify for extra funds if they have foreign students • CH Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)

Why Admit? Key objective: Institutional Profile International profile is of intrinsic value for Institution • Foreign students enhance innovation and modernization • Global pool better for choosing world-scholar quality • Minimum or maximum target % may be set: AU • Global recognition of host university for its student quality is very competitive • Competition at level of Institution/ Discipline or at level of Labour Market Demands • Nature of programme requires international approach • Research literature is in English (MBA, Engineering): FR • Applied Languages, International Relations, International Law • Foreign students bring the world home to remote areas • AU, SE Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)

Why Admit? Key objective: Sustainability Securing the future of the programme, staff, research • Smaller countries do not have enough local students for keeping some disciplines alive at their universities • NL, SE, CH need Science students • Students coming for Masters programmes are encouraged to stay on for PhD programmes to secure enough teaching staff assistants and research assistants Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)

Why Admit? Key objective: Non-discrimination No discrimination is tolerated between any student groups • Quality of students comes first regardless of student’s origin • Important objective for highly selective / private institutions: US • Some are more equal than others (informally or formally) • CH: evidence required that foreign student could have studied back home • FR: to protect local student quota, local students chosen with earlier deadline for public university places • SE: first choose local students and unfilled quota goes to foreign students Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)

Why Admit? Key objective: Students’ Demand Large number of foreign students want to come to the Institution • Local, regional and global mobility (economic, academic, professional) is a growing demand by students, governments, labour market so students expect international programmes in countries of interest to them • Universities receive large number of foreign student applications and react by starting international programmes and (hopefully) support programmes Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)

How to Admit? Look for QualityWhat is Student Quality? ACA research showed a variety of instruments for gathering evidence on student quality • Secondary school/ Bachelor diploma • Transcript of courses • List of completed courses • Motivation statement/personal statement • Letter of recommendation • Proof of work experience • Proof of previous academic work • Curriculum Vita • Standardized tests for subject and skill competencies • Proof of knowledge of the teaching language • Right to access Higher Education in home country Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)

How to Admit? Look for QualityWhat is Student Quality? College Board research on factors used in making admissions decisions • Academic Achievement, Quality, and Potential • Direct measures (coursework, grades, test scores) • Caliber of high school/ previous study institution (rank of class, school) • Evaluative measures (depth and breadth of interests, world awareness, communication skills) • Non-academic Characteristics and Attributes • Geographic (remote or disadvantaged region, local resident) • Personal Background (minority, socio-economic status, age) • Extra-curricular activities, service, & leadership (awards, social service, work experience) • Personal attributes (talent, character, commitment, social concern, creativity, determination) • Extenuating circumstances (family problems, health issues, language difficulties) This may differ somewhat from what institutions tell students what they are looking for Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)

known inclusive exclusive unknown University policy for educating students Where do Student Quality Instruments fit? Student background Low score on TOEFL ITP or TOEIC High score on GMAT High score on TOEFL

known inclusive exclusive unknown University policy for educating students Where do Student Quality Instruments fit? Student background • Motivation statement/personal statement • List of completed courses • Letterofrecommendation • Proof of work experience • Secondary school/bachelor diploma • Right to access HE in home country • Proof of knowledge of the teaching language • Proof of previous academic work • Standardized tests

capable refused admitted notcapable How to decide depends on which type of error you tolerate more

known inclusive exclusive unknown Consider exceptional cases:(a) Lottery systems (Numerus Clausus)(b) Admissions and immigration (NL, DE) University policy for educating students Student background

‘ Capable’ = Successful How do we recognize success? Popular indicators of evidence: • Student quality • Attrition III. Time to degree • Faculty ratings • Grade point average • Results of Exams

Student Quality = Some qualities are also important to the labour market Competencies important for graduate school success Which Student Quality?

Student Quality = Student Quality? Competencies important for graduate school success Three research studies • Two studies asking faculty for competencies of successful students • One scheme for classifying outcome or criterion variables: “taxonomy of higher order performance components.”

Types of Competencies linked to Student Quality • Job-specific, Study-specific task proficiency • Non-job, Non-study-specific task proficiency • Written & Oral Communication task proficiency • Demonstration of effort • Maintenance of personal Discipline • Facilitation of peer and team performance • Supervision/ Leadership • Management/ Administration • Some are present when student is admitted • Some are part of the curriculum that the institution provides (educational value added) and so are expected at graduation Source: Standardized Letter of Recommendation, Educational Testing Service

Competencies • Job/study Specific Task Proficiency 1. Research/scholarly experience (amount and quality) 5. Mastery of discipline 11. Ability to read and analyze research in the field 13. Breadth of perspective C. Explanation (Enright & Gitomer) G. Synthesis (Enright & Gitomer)

Competencies II. Non-Job/study specific Task Proficiency 7. Ability to teach (cross-listed in VII and VIII) 8. Independence (cross-listed in IV) 9. Creativity; ability to think “out of the box” 14. Open mindedness 15. Opportunism/Resourcefulness 16. Critical thinking ability, logic, problem solving ability 18. Computer literacy/Understand & Manage Technology 19. Skill in investigation 20. Ability to find the facts: information gathering B. Creativity (Enright & Gitomer) E. Planning (Enright & Gitomer)

Competencies III. Written & Oral Communication Task Proficiency 4. Communication (professional) 6. English-as-a-Second-Language ability A. Communication (Enright & Gitomer) IV. Demonstration of Effort 2. Persistence/Tenacity* 8. Independence* 10. Enthusiasm* 17. Drive/Commitment/Motivation/Zeal* D. Motivation (Enright & Gitomer)

Competencies V. Maintenance of Personal Discipline 12. Values/Character: integrity, fairness/openness/honesty, trustworthiness/ consistency – personal and professional* 21. Professional posture VI. Facilitation of Peer and Team Performance 3. Collegiality/Making professional connections/Networking (Cross-listed with VII) F. Professionalism VII. Supervision/Leadership 3. Collegiality/Making professional connections/Networking (Cross-listed with VI.) 7. Ability to teach (Cross-listed with II. and VIII.) VIII. Management/Administration 7. Ability to teach (Cross-listed with II. and VII.)

Cross reference to Tuning Project How are competences and learning outcomes related? • Learning outcomes according to Tuning methodology should be formulated in terms of competences. • Learning outcomes are requirements of a unit or a programme and are expressed in terms what the learner knows and is able to do at the end of the learning experience. • Competences may be developed to a greater degree than the level required by the learning outcome.

Staff-centred vs. Student-centred?

On completion of a first cycle degree in Mathematics, students should be able to • Show knowledge and understanding of basic concepts, principles, theories and results of Mathematics • Understand and explain the meaning of complex statements using mathematical notation and language • Demonstrate skill in mathematical reasoning, manipulation and calculation • Construct rigorous proofs • Demonstrate proficiency in different methods of mathematical proof.

How to recognize competencies?

The Admission Challenge What are the competencies you attribute to successful students? Which of them have to be present at the start? Is the aim Inclusive or Exclusive selection?

The Admission Challenge What evidence do you consider:Is it fair, valid, reliable? Have you considered all costs? What is the legal setting?

Evaluating an Application: Different models of meeting the Admissions challenge Admissions offices may be responsible for • Combining all individually-rated components and weighing • Application information • Transcripts • Essays • Recommendations • Combining academic and non-academic factors and weighing • Some factors considered separately e.g., communication • Balance of academic and non-academic • Many wide-scope factors vs. few key factors Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)

Evaluating an Application: Different model examples Example 1 • Academic • Personal characteristics • Suitability for desired field of study Example 2 • Academic • Communication • Character, leadership, initiative Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)

Evaluating an Application: Different model examples Example 3 • Exceptional academic Achievement • Academic Promise • Potential to Contribute Example 4 • Academic Achievement • Academic Qualities • Non-academic Achievement • Personal Qualities Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)

Evaluating an Application: Different model examples Example 5 • Quality of Courses • Grades in Core Curriculum • Test results • Activities (extra-curricular or curricular) • Essay Example 6 • Academic Performance • Extra-curricular activities • Teacher / counselor recommendations • Interview • Personal Inventory • Essays Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)

Evaluating an Application: Different model examples • Example 8 • Academic Achievement • Intellectual Curiosity • Potential • Commitment • Communication • Engagement with others • Extra-curricular activities • Initiative Example 7 • Application and Essay • Academic Performance • Level of Challenge of Academic Record • Recommendation and Interview • Personal Qualities • Special Talents Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)

Making Decisions: Making sense of what information comes in from Applications • Single readers, multiple readers, group readings, automated readings • Process x for collecting academic info (formulaic rating) into an index • Process y for collecting non-academic info (holistic rating) into an index • Clear minimum thresholds of entry per index or factor • Clear criteria for strong, and weak candidates • Level descriptors for each level of qualification • Clear guidelines for multiple entryways • Special consideration for minorities, shortage of students • Lottery for some competitive programmes with limited places • Work experience • Final review of denials • Conditional acceptance • Additional information or interview requested

Quality Assurance for Application Models • Setting key current criteria into application form • Direct and indirect questions • Setting dates for re-evaluation of application form criteria • Selection of staff who represent key constituents affected by decision • Adapting admissions criteria to institutional profile over time • Respecting changes in strategic objectives • Adapting to external changes in education, demographics, labour market • Understanding where the student body comes from and goes to

Best Practices in Admissions What to look for Qualities sought in student body How to Select AdmissionsCriteria Consensus on Admissions objectives How to Decide • Application & Processing models • Fairness • Validity

Key Messages • Admissions Criteria are the outcome of a decision process which must be regularly evaluated • The process has the following components: • Strategic objectives of the institution • Instruments to identify student quality • Student application model • Admissions criteria

Discussion and insights • Why do you admit students? • What are the 3 main Strategic Objectives of your institution? • What are your main 3 Admission objectives? • What are your 3 most common profiles of admitted students? • 2. What is student quality for you? • What are the 3 main student-quality instruments that you use? Consider how your policy objectives affect the level of detail that you require from your instruments. • What are the most common instruments of your peers? • Where do your admissions instruments fall on this chart?

known inclusive exclusive unknown Where do your Student Quality Instruments fit? University policy for educating students Student background

3. What is the quality of your added value to the student’s education? • Which 2 key competencies does your institution expect to be present at admission? • Which 2 key competencies does your institution expect at the learning process? • 4. How do you recognize quality? • What evidence does your institution require? • Evidence from the student directly or other institutions • Student portfolio • High-school/ previous programme transcripts • Letters of recommendation • Personal answers to questions on the application • Candidate’s scores on standardized admissions tests • Language standardized tests • Which formal instruments do you use? • How do you ensure fairness, validity, and reliability? Discussion and insights

Discussion and insights • 5. What type of student application do you use? • What are the top 3 factors in your qualification evaluation model? • What quality assurance principles to you use to keep your application model valid? • How frequently do you hold re-evaluation? • How do you ensure validity of criteria/ benchmarking success factors • What is your process of updating criteria?

Best practices in continuous evaluation: Evidence-Based Decision Making and Admissions Models