EduDDI – an application of DDI 3.x for large scale assessments in education

1. EduDDI – an application of DDI 3.x for large scale assessments in education IASSIST/IFDO 2009 Tampere, Finland May 26th-May 29th, 2009 Martin Mechtel, Dr. Michel Knigge

2. Outline • IQB – Institute for Educational Progress • Expected benefits of meta data standardisation • Efforts so far (drawbacks) • Introducing DDI 3.x (bridges) • Road map of EduDDI (sorry no software demo)

3. IQB – Institute for Progress in Education • Founded 2004 by the Bundesländer (Federal States); hosted by Humboldt-University at Berlin, Principal Prof. Dr. Olaf Köller • Trigger: TIMSS 1995/PISA 2000; frequent large scale assessments of students competencies take place on a regular basis since then • Mission: Test educational standards for Germany • Staff: Meanwhile 40 plus 40 part time (students) • Budget: About 4.5 Mio € per year, plus extra projects (e.g. FDZ – research data center) • My role: Responsible for IT services (infrastructure, development); IT-staff: 3 plus 4 part time collegues (most infrastructure is supported by Humboldt-University) • Dr. Michel Knigge: Head of FDZ

4. Expected benefits of meta data standardisation Obvious: • Interoperability: not only "meta data" but datasets too • Exchange of items/tasks easily, support of re-analyses, nice presentation of data to the public etc. • Thinking: when I have a standard then I have tools which do the work for me Side effects: • Installation of structures, work flows and vocabularies • Dissemination of best practises (statistical methods, software for analyses, reporting culture, access policies)

5. Efforts so far (drawbacks) – specific in education 1 • Item appearance is crucial for interpretation of results (line breaks, font, size etc.) • Item structure is complex (task: stimulus, sub tasks, items for scoring, variables for analyses)

6. Vogonische Lyrik Task M042 Grupp, ich beschwöre dich, mein punzig Turteldrom Und drängel reifig mich mit krinklen Bindelwördeln Denn sonst werd ich dich rändern in deinen Gobberwarzen Mit meinem Börgelkranze, wart´s nur ab! Stimulus Aufgabe 1 Welches Sprichwort passt am ehesten zu der Aussage des Gedichts? Kreuze die richtige Antwort an.Reisende soll man nicht aufhaltenLiebe macht blindOhne Fleiß keinen Preis Nachts sind alle Katzen grau. Subtask A Item 01 Variable a An welcher Zeile ist das erkennbar (Zeilennummer): Item 02 Variable a Aufgabe 2 Nenne zwei Klangfiguren in diesem Gedicht: Subtask B Item 03 Variable a Variable b Efforts so far (drawbacks) – specific in education • item appearance is crucial for interpretation of results (line breaks, font, size etc.) • item structure is complex (task: stimulus, sub tasks, items for scoring, variables for analyses)

7. Efforts so far (drawbacks) – specific in education 1 • Item appearance is crucial for interpretation of results (line breaks, font, size etc.) • Item structure is complex (task: stimulus, sub tasks, items for scoring, variables for analyses) 2 • Complex study design to estimate the level of a certain competence although the test person responds simply to one or two items of the scale • Item position is important for interpretation

8. test design (example) booklets timeline total 167 Min Efforts so far (drawbacks) – specific in education

9. 3 development of a standard failed Efforts so far (drawbacks) – specific in education 1 • Item appearance is crucial for interpretation of results (line breaks, font, size etc.) • Item structure is complex (task: stimulus, sub tasks, items for scoring, variables for analyses) 2 • Complex study design to estimate the level of a certain competence although the test person responds simply to one or two items of the scale • Item position is important for interpretation 3 • Mostly small institutions involved in assessment (EMSE – Bundesländer; isolated chairs in research); like simple procedures • Mostly short-term projects; no pay-back of investment in comprehensive approaches

10. Vogonische Lyrik Task M042 Grupp, ich beschwöre dich, mein punzig Turteldrom Und drängel reifig mich mit krinklen Bindelwördeln Denn sonst werd ich dich rändern in deinen Gobberwarzen Mit meinem Börgelkranze, wart´s nur ab! Stimulus Aufgabe 1 Welches Sprichwort passt am ehesten zu der Aussage des Gedichts? Kreuze die richtige Antwort an.Reisende soll man nicht aufhaltenLiebe macht blindOhne Fleiß keinen Preis Nachts sind alle Katzen grau. Subtask A Item 01 Variable a An welcher Zeile ist das erkennbar (Zeilennummer): Item 02 Variable a Aufgabe 2 Nenne zwei Klangfiguren in diesem Gedicht: Subtask B Item 03 Variable a Variable b Introducing DDI 3.x (bridges) But now: we HAVE a standard! "EduDDI"

11. Introducing DDI 3.x (bridges) • DDI can do: specific structure, other material, scales • DDI should do (3.2?): usage of Genericode to attach any property to any XML item (inline storage of data to get a question bank) • Working with DDI expands the community (by YOU) • DDI is the cheese for the mouse: possible partners are soon convinced and support the project Other fortunate conditions: • New high stake/large scale project looks for technology: NEPS National Educational Panel Study • In Germany, a sufficient number of large studies wait for further analyses (FDZ)

12. Road map of EduDDI Project/initiative members: DIPF Frankfurt/M., TestDaF Hagen, IEA-DPC Hamburg, IQB+FDZ, ifib Bremen, University of Bamberg (NEPS) Near future • Discussion about meta data catalogues (two groups: item oriented and dataset oriented) • Discussion about technology/architecture (inside institutions: ASP/VB.Net, MS SQL Server; offline tools: java) • Acquirement of more support (project proposals, more institutions) Next year • Definition of the DDI implementation • Development of first tools and interfaces • Launch of research studies to underpin the assumptions and to bring the project to a broad community

13. telefon +49[30]2093-5335 telefax +49[30]2093-5336 IQBoffice@IQB.hu-berlin.de www.IQB.hu-berlin.de postal address Humboldt-Universität zu Berlin Unter den Linden 6 10099 Berlin Location Luisenstr. 56 10117 Berlin