E-Assessment in Denmark – E-exams and CAT as a Pedagogic Tool Jakob Wandall, Chief Adviser,

1. Tallinn, Estonia 14’th October 2010 E-Assessment in Denmark – E-exams and CAT as a Pedagogic Tool Jakob Wandall, Chief Adviser, Skolestyrelsen (Danish National School Agency)

2. Background Test and exams in Denmark The Exams The story behind the testing system The design of the adaptive system The differences between danish tests and exams Challenges & Future possibilities Danish Assessment in schools

3. Background: The Danish School system

4. Facts of the public primary and lower secondary education in Denmark, ”Folkeskolen”: 600.000 pupils, 60.000 pr. form 1.300 ordinary folkeskoler, 600 other institutions (e.g. for special education) School governed by decentralized local governments  significant differences from municipality to municipality 4-5 pupils pr. new computer (<3 year) highspeed internet-connection almost everywhere  Infrastructure suited for It-based assessment Background: the Danish School system

5. Teachers in Denmark Until 2004: No common objectives between 1. and 9. form in “Folkeskolen”  significant differences from school to school Poor tradition for standardized assessment of pupils – e.g. weak tradition for testing and use test results Strong culture for independent, self-governing teachers with focus on “soft evaluation methods” Weak tradition for leadership in school Anxiousness of the control aspect in the test system. Background: the Danish School system

6. Types of assessment Formal – provided by the MoE Final exams in different subjects by the end of 9’th form National tests in different subjects/forms Informal - Locally provided Tests and other kinds of assessment and evaluation Background: the Danish School system

7. Tests & Exams- in the Danish School system

8. Long tradition for exams Preparation for confirmation in 1736 – School was provided by the church First exams for school, that were defined by law in 1814. Decree about peasants-school (precurser for the act of Folkeskolen) Revision of examination regulations in 1899, 1903, 1937, 1958, 1975, 1993 og 2006 1975 - 2006 exams were optional – mandatory from 2006, with a chance for dispensation. Weak tradition for using test Increasing use of test during the last decade National test introduced in the legislation in 2006

9. Final exams in the end of form 9 compulsory in public schools from 2006 Renewed, modernized and increased in number (more subjects and some E-based exams from 2005) Ensuring a better foundation for completing a post-compulsory education School leaving exams • Each student must sit for minimum 9 exams • Assessed by external examiner and the teacher • No failing criteria – but formally qualifying - requirement for further education • Aggregated results are published for each school • Summative evaluation, High Stake

10. 3 types of exams: Verbal/practical exams (7) Written exams – eg. Essays (2 + 3) Standardised test exams – including 2 E-based (5 + 3) School leaving exams in form 9

11. E-based exams has been administered in science (Biology and Geography) since 2005 Delivered via internet (on-line) Linear format – 20 tasks, 50-60 items, ½ hour Traditional types of Items - Mostly multiple choice Not tested on pupils Large diversity in contend (no IRT) E-based standardised exams

12. Example: Biology – summer 2009

13. Example: Biology – summer 2009

14. Example: Geography – summer 2009

15. The scale comparable with ECTS-scale 12 = A 10 = B 7 = C 4 = D 2 = E 00 = F -03 = Fx Activity: 700.000 exams in 2008/09 High rate of completion: Completed: 98,3% Exempted: 0,7% Absentees: 1,0% Exams: scale and activity

16. The National testing system - atool for improving the culture of evaluation

17. PISA-surveys (2000-2006) OECD-review on Denmark (2004) and National reports on assessment in school (2004) Governments initiatives according to the OECD-Recommendations Background for the Danish National tests

18. The Danish PISA results, 2000 - 2006 OECD-average 2006

19. Review of National Policies for Education, Denmark, OECD 2004Denmark has one of the most expensive education systems in the world, and for years perceived it to be one of the best in the world. However the disappointing results of recent international tests to measure schooling outcomes confirmed earlier evidence, that the system actually is underperforming. ” ”

20. Strengths, weaknesses, 35 recommendations Some weaknesses: Poor tradition of pupils assessment, Insufficient teacher qualifications in assessment techniques Insufficient exchange of best practices between teachers Some recommendations for the Minister: Development of criteria-based tests. Development of different assessment methods and -materials. Carry out a policy based on the principle, that test results don’t get published for ranking purposes. OECD-review 2004

21. Government initiatives to improve the culture of evaluation

22. Test and assessment systems in some countries

23. Reference countries to Denmark

24. The Government initiatives followed the OECD recommendations with some modifications – e.g. in the testing system The OECD-team recommended criteria-based test - recommendation based on a different tradition of testing/evaluation Background: The OECD-team came from England, Canada and Finland Implementation of OECD’s recommendations

25. Two different traditions for assessment/use of test results • The Nordic / continental Terminology, Nordic/German origin (Danish: Prøve, vurdering, bedømmelse, opgave, karakter) 2. The Anglo-American Terminology, English origin (Danish: test, score, item)

26. Traditions for assessment/testing, Why testing? What are the results used for?

27. Traditions for testing, Why test pupils – What are the results used for Pedagogy/Teaching Analysis  Teacher Learning  Pupil The purpose of testing The use of the test results Control/financing

28. Test in combination with other assessment tools Easy to use for the teacher Flexible systems from the schools point of view Low/no cost for the school Priority to pedagogical purposes – Formative – Low stake - For teachers assessment of the pupils - The teacher sets the rules Effective Self-correcting tests valid, reliable detailed results Max. 45 minutes (1 lesson) Central administered internet-based Computer adaptive testing - with focus on pedagogy Criteria for choice of strategy

29. The Danish testing system

30. The national tests Forms where tests can be used X 10 compulsory tests

31. IT-based, automatically scored, provided freely by the National School Agency. Pedagogical purpose  The teacher sets the rules for the test The teacher are booking & administering the tests and interpreting the test results Different feedback on various levels (teacher, headmaster, municipality) The teacher gets detailed results online for the class, the individual pupil and Details of the tests, including the items in the individual tests. Main features of the Danish national tests

32. Adaptive = Adapts to the individual pupils ability: Correct answer  More difficult questionsWrong answer  Easier questions The test is most efficient when item difficulty = pupil ability More effective testing  more detailed results: Adaptive within 3 “profile areas”  3 tests in 1 (E.g. English: Reading, Vocabulary and Language usage) Simple principles, but a few tricky conditions: Extensive demands to the technology – both capacity and stability. Very large Item-pools with the exact right mix of high quality items CAT (Computer Adaptive Testing)

33. Condition for adaptive testing is: That the difficulty of the items are well defined and stabile over time (homogeneity). No differential item function. Sufficient number of items, so that even the fastest pupils don’t run out of items That the items are evenly distributed on item-difficulty  challenges for all pupils. Therefore following requirements have to be met: Minimum 540 items pr test (180 pr. Profile area) – evenly distributed on difficulty levels All items are tested on a large number of pupils (500-700) The items are required to fit a Rasch model Not more than 3 runs pr pupil/test (including compulsory test) Creating the item pools

34. An exampel: Adaptive test – English reading Item difficulty / pupil ability level Item number

35. Adaptive test – English reading ≤ 5 items pr. profile area > 5 items & SEM > 0,3 SEM ≤ 0,3 Item difficulty / pupil ability level Item number

36. Adaptive test – 3 profile areas Item difficulty / pupil ability level Item number

37. Different feedback on various levels (parents, teacher, headmaster, municipality) The teacher gets detailed results online for the class, the individual pupil and Details of the tests, including the items in the individual tests. The test results

38. Presentation of the results:Only online for teachers (no print-outs) Test results (average of pupils results) from the class (translated by Google)

39. Presentation of the results: Graphical illustration of the class results

40. Presentation of the results: Results of the individual pupils and the class

41. Presentation of the results: Details of the individual pupils

42. Presentation of results:A letter for the parents

43. Presentation of the results: Details of the individual pupils

44. Presentation of the results: Detailed information on the pupils test course

45. Presentation of the results: View the individual responses

46. Status on the implementation:Lesson learned – it takes time! • The first 3 test was launched 2007, with reduced item banks • There has been conducted an expert review that showed, that • the psychometric standard was at a very high level, but • the quality of the items and size of the item pools were insufficient • Consequence: The planned full scale launch march 2008 vas delayed – instead - Development and testing of items in 2008 and 2009 • Full scale tests of the test system in autumn 2009 • Test system launched march 2010 Contract signed Contract prolonged The first 3 test launched Test system launched Development and testing of items Review Website launched 2011 2010 2007 2008 2009 2009

47. Folkeskolen’s final exams High-stake, criteria based, goal-orientet, mixture of standardised and non-standardised exams. ”Teaching to the test” not a problem, as test-objectives are similar to the objectives for the teaching (Final national Common Objectives). Precise rules for how to conduct an exam - same items, same time, same conditions – eg. aids ect. – for all pupils. Formal documentation of the education – school results are published The differences between tests and exams

48. National tests Low-stake, norm referenced, standardised proficiency test with a diagnostic element, which can be used as progress / added value testing. The teacher decides almost everything – when, how, where, how long time, how many items etc. Internal assessment tool Prioritised purposes: 1: Pedagogic tool for teachers and headmasters – Only access for those who need the results 2: Documentation for the academic level The differences between tests and exams

49. Items look alike, but their purpose are different: Main goal for exams is to provide documentation with legal effect. Main goal for tests is to provide an assessment of the student’s academic level for pedagogical purposes. General requirements for items – distractors must be both plausible and unambiguous : Examination items: Focus on unambiguous (one correct answer!) –accessible for public / the right to complain Test items: Focus on plausibility. (more or less correct answers – gives challenges to pupils  more difficult questions). Quality control Exams are designed by exam commissions – skilled members with teacher background and long experience Test items are written by experienced teachers, quality controlled by experts, tested on 5-700 pupils and fits the Rasch model Exam- and test items

50. Present challenges & Future possibilities