New Bulgarian University Assessment Center

1. New Bulgarian University Assessment Center Maurice Grinberg www.nbu.bg evaluation.nbu.bg ALTE meeting, Lisbon, November 12-15

2. New Bulgarian University Modern private university Year 1991: 4 000 students in 30 programs Year 2008: 15 000 students in more than 100 programs Bachelor, Masters, and PhD programs Distance learning For the first time in Bulgaria:General Education Test as entrance exam four times a year Assessment Center ALTE meeting, Lisbon, November 12-15

3. NBU Assessment Center • Since 1995: General Education Test as admission exam (more than 10 000 candidates per year) • Since 1998 for all NBU students: • L2 exams at the end of a course of study, levels from A2 to B2 • 6 languages: English, German, Italian, Russian, Spanish, and French • L1 exam for writing skills in Bulgarian language • Computer skills exam • Since 2002: placement tests ALTE meeting, Lisbon, November 12-15

4. NBU Assessment Center • Staff: Experts in testing, psychometrics, cognitive science, languages • Advisory board: • Department of Psychology and Cognitive Science, and department of Foreign Languages • NBU QA department • External consultant: prof. D. Dimitrov, George Mason University • Projects and research • Leonardo da Vinci program (WELKOM) • Erasmus program (NETTLE) • Assessment in e-learning • Cognitive assessment, small group assessment ALTE meeting, Lisbon, November 12-15

5. NBU Assessment Center Monitoring and development of all examination procedures in NBU Development of examsfor NBU students Analysis and interpretation of test results: test score reports to test takers and university administration Item-banking and storage of data Training for item-writers and standardized workshops for examiners ALTE meeting, Lisbon, November 12-15

6. NBU Assessment Center • Fairness and equity • Standardized administration • Monitoring during examination • Special accommodation for candidates with disabilities • Information about the content and format of the examination: question types, rubrics and appropriate test-taking strategies, test samples ALTE meeting, Lisbon, November 12-15

7. Second Language Exams at Level B2 • B2 is a required level in NBU for all students • Goals: use the language to express and understand opinions, to build up and understand arguments, to communicate with students from other countries in English, Spanish, Italian, French, German or Russian. • Certification decisions and graduation ALTE meeting, Lisbon, November 12-15

8. Second Language Exams at Level B2 Test takers: Bulgarian native speakers with different topical knowledge and second language experience Tasks: both from the instructional and real-life domain Scores are reported to candidates in grades Tested abilities: speaking, listening, reading, and writing ALTE meeting, Lisbon, November 12-15

9. Second Language Exams at Level B2 • Psychometrics: • IRT used for evaluation of test item properties • Many-facet Rasch analysis in assessing essay writing task (facets: topic, raters, and assessment criteria) • Estimation of candidates’ abilities, raters’ severity, relative difficulty of tasks, and relative difficulty of assessment criteria • Bias/interaction analysis ALTE meeting, Lisbon, November 12-15

10. Goals of NBU AC Goals of NBU AC in Associating with ALTE • Meet all ALTE standards for quality in language tests: • Revise existing foreign language exams • Achieve a consensus about test qualities • Develop a new test for Bulgarian as a foreign language • Exchange of best practices: • Investigation and assurance of test reliability • Improvement of validation argumentation • Elaboration of procedures to involve all stakeholders in the examination processes ALTE meeting, Lisbon, November 12-15

11. Goals of NBU AC in Associating with ALTE • Development of tests for Bulgarian as a foreign language • Situational analysis: need for a A1 test for foreign university students in Europe and Asia and related textbooks • Planning phase: users and test takers identified, resource balance performed ALTE meeting, Lisbon, November 12-15

12. Goals of NBU AC Goals of NBU AC in Associating with ALTE • Development of tests for Bulgarian as a foreign language (cont.) • Design phase: theoretical background of the test • Approaches considered: • Bachman model of CLA will underlie each task as a needed executive resource • CEFR for the domain, communicative goal and language function • Weir’s Frameworks for Developing and Validating tests ALTE meeting, Lisbon, November 12-15

13. Thank you! NBU: http://www.nbu.bg Assessment center: http://evaluation.nbu.bg(in construction!) ALTE meeting, Lisbon, November 12-15

14. Achieving overall usefulness of the examination: problems and decisions About Impact: Our strive to achieve positive impact involves Providing public information and advisory services for students Seminars to familiarize university teachers with our research on test qualities Training for item writers and examiners Reports to the University Board Cooperative work with the Department of Foreign Languages to improve continuous assessment in language classes Problems: we have not investigated the educational impact of our examinations Decisions: start collecting evidence about “face validity” About Practicality: The Assessment Center develops and implements the appropriate procedures for managing all aspects of the examination process. Problems: need to increase research studies in all aspects of test production Decisions: identification of suitable experts ALTE meeting, Lisbon, November 12-15