AEA-Europe 2013 Paris Fabienne Van der Kleij Jorine Vermeulen Kim Schildkamp Theo Eggen

1. Data-BasedDecisionMaking, AssessmentforLearning and DiagnosticTesting in FormativeAssessment AEA-Europe 2013 Paris Fabienne Van derKleij JorineVermeulen Kim Schildkamp Theo Eggen

2. Formative assessment • …lack of a uniform definition

3. Formative assessment is an umbrella term informed teaching progress self-regulation peerassessment reflection authentic rubrics responding to data observation feedback targeted instruction classroom dialogues diagnosis understanding attainingstandards student ownership

4. Evaluation or assessment • Different aggregation levels Formative evaluation the use of assessment data to make decisions concerning the quality of education at a higher aggregation level than the level of the learner or the class Formative assessment concerns decisions at the levels of the learner and the class to accommodate the pupils’ individual educational needs

5. Focus on the quality of the learning process • Three approaches to formative assessment Attaining results and targets Data-based decision making DBDM Assessment for learning AfL Diagnostic testing DT Detailed insights into learning processes and outcomes

6. Educational practice implements a mix of these approaches. • Terminology and definitions are inappropriately used interchangeably  theoretical ambiguity. • Discussions tend to revolve around finding evidence of what works.“quick fixes” (Elwood, 2006). • “Definition is important because if we can’t clearly define an innovation, we can’t meaningfully document its effectiveness” (Bennett, 2011, p. 8). • Need for prescriptions on why, how, and when assessment should be used by learners, teachers, and schools to enhance learning. • Three approaches to FA • … lessons from the literature on formative assessment

7. Initial exploration: Recognise the fundamental differences amongst these approaches  blend these approaches in a meaningful way. • Contribute to a more coherent research agenda. Note: We do not intend to make any claims about which assessment approach is most effective. • Aims and scope • Comparative theoretical study

8. What are the similarities and differences in the theoretical underpinnings of DBDM, AfL, and DT? • What are the consequences of these similarities and differences for implementing DBDM, AfL, and DT in educational practice? • Research questions

9. Starting point for implementing a system-wide formative assessment approach: An understanding of the learning theories behind currently dominant approaches (Elwood, 2006) • Neo-behaviourism • Cognitivism • Meta-cognitivism • Social cultural theory • (Social) constructivism • Learning theories and feedback

10. Neo-behaviourism: Introduction of the formative concept. memorisation of facts, feedback is intended to reinforce correct recall of facts. • Cognitivism: Highlights information processing and knowledge representation. Feedback is intended to correct incorrect responses. • Meta-cognitivism: Emphasis on learning how to learn and regulating the learning process. Feedback is about how to learn, rather than about what to learn. • Social cultural theory: Feedback in the form of scaffolding (ZPD) for acquiring knowledge and skills, focus on what students are able to learn. • (Social) constructivism: Learning is a cyclic process in which new knowledge and skills are continuously built on prior ones. Active engagement, realistic contexts, learning in interaction, frequent elaborated feedback. • Learning theories in a nutshell

11. Theoretical underpinnings and implementation Data-based decision making DBDM Assessment for learning AfL Diagnostic testing DT

12. Schildkamp and Kuiper (2010): "systematically analyzing existing data sources within the school, applying outcomes of analyses to innovate teaching, curricula, and school performance, and, implementing (e.g., genuine improvement actions) and evaluating these innovations" (p. 482). • Data in the context of schools is "information that is systematically collected and organized to represent some aspect of schooling" (Lai & Schildkamp, 2013, p. 10).  e.g., assessment results, student background characteristics, questionnaire results. • Theoretical underpinnings • DBDM

13. Process of data use in DBDM • Feedback loop Identify data Data use Interpret data Collect data Analyse data

14. Early initiatives: Based on neo-behaviourism and cognitivism. • Move towards social cultural theory and constructivism. • DBDM for formative assessment: • Set learning goals, assess and monitor student achievement, and if necessary, adjust instruction. • DBDM for formative evaluation • Policy development, school improvement planning, teacher development, and monitoring the implementation of the school's goals. • DBDM • Learning theories and processes

15. Aggregation levelSchool, class, and student levels. • Assessment methods Objective data from standardised tests, e.g., from a PMS. Formally gathered data.  Less frequently available. + Data from standardised assessments and (structured) observations. • Feedback loops Feedback based on assessment data. Educators  transform data into actions for educational improvement. Length improvement cycles and feedback loops varies. Retroactive in nature. • Implementation of DBDM • Aggregation Level, Assessment Methods, and Feedback Loops

16. Theoretical underpinnings and implementation Data-based decision making DBDM Assessment for learning AfL Diagnostic testing DT

17. ARG "… the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and how best to get there" (2002, p. 2). • Klenowski (2009) "part of everyday practice by students, teachers and peers that seeks, reflects upon and responds to information from dialogue, demonstration and observation in ways that enhance ongoing learning" (p. 264). • Hargreaves (2005): two approaches within AfL • Theoretical underpinnings • AfL Measurement Inquiry

18. Underpinned by meta-cognitivism, social cultural theory, and social constructivism. • Learning is a social activity; learning occurs through interaction. • Students have a central role in the learning process; active participation. • Increase learner autonomy, motivation, and reflection, by facilitating an inquiry-oriented and interactive classroom climate. • AfL • Learning theories and processes

19. Aggregation levelClassroom and student levels. • Assessment methods Paper-and-pencil tests, dialogues, practical demonstrations, portfolios, peer assessment, or self-assessment. Qualitative & quantitative. Planned & unplanned. Formal & informal. Assessment quality  actionable information for formative purposes over the short term. • Feedback loops Continuous dialogues and feedback loops, (immediate) feedback to direct further learning. Assessments integrated into the learning process, feedback loops are usually short.  continuous adaptation. Interactive, sometimes retroactive or proactive. • Implementation of AfL • Aggregation Level, Assessment Methods, and Feedback Loops

20. Theoretical underpinnings and implementation Data-based decision making DBDM Assessment for learning AfL Diagnostic testing DT

21. How a task is solved is indicative of the developmental stage of the learner. • Identify the learner's reasoning styles, and skipped or wrongly executed procedural steps caused by misconceptions and prior knowledge. • Principles from cognitive psychology, subject pedagogy, and learning theories are combined. inferences about student learning based on a student's response patterns. • Theoretical underpinnings • DT

22. Underpinned by principles from cognitivism. • Vygotsky's social cultural learning theory  ZPD. • Identify the learner's developmental stages by obtaining action-oriented, fine-grained assessment data (process data). • Cognitive theories are used to interpret process data to identify misconceptions and knowledge associated with the learner's developmental stage. • Small-grain size of the measurements in DT  exceptionally useful for formative purposes. • DT • Learning theories and processes

23. Aggregation levelStudent level. • Assessment methods Assessment tasks  make inferences about how task behaviour relates to student thinking. Inferential chain from empirical knowledge from information processing theories, cognitive psychology, and learning trajectories. Assessment results: indicative of the developmental stage. Use of CAT and dynamic assessments. • Feedback loops Short feedback loops in are preferred. Retroactive, sometimes interactive or proactive. • Implementation of DT • Aggregation Level, Assessment Methods, and Feedback Loops

24. Comparison of the three approaches • Theoretical underpinnings of DBDM, Afl, and DT

25. Aggregation levels • Assessment methods Diversion in the goals: large variety in assessment methods  associated with the nature of data, purposes and stakes regarding the use of these data. • Feedback loops Take place at various levels and frequencies. • Comparison of the three approaches • Implementation of DBDM, AfL, and DT Assessment Student Class School Evaluation DBDM AfL DT

26. Original theoretical underpinnings of the approaches differ. • Various assessment methods that are underpinned by different learning theories are needed to fully grasp the complexity of learning at all levels. • Integrating the three assessment approaches can lead to more valid formative decisions. • Potential to enable school staff to continuously provide feedback at the school, class, and individual levels, to guide and enhance student learning. • Conclusion • Differences, similarities, and possibilities

27. DBDM: School, class and student levels: Monitor learning goals, group students, improve the quality of education.  Feedback loops extend over a longer time span. • AfL: Class and individual levels: Improve quality of the learning process by engaging learners. Continuous feedback in short cycles. • DT: individual level: Collect fine-grained data that can inform decisions on adapting the learning environment.  Feedback loops occur irregularly; depends on learner’s needs. • At different points in the education process, retroactive, interactive, or proactive feedback loops can be used to optimise students’ learning processes. • Conclusion (continued) • Blending the approaches

28. Towards a balanced assessment system • Alignment of formative with summative practices “The field will need to marry information from the different assessments to produce a union that a stronger than any of the parts in isolation” (O’Malley, Lai, McClarty, & Way, 2013, p. 146).

29. Thankyouforyour attention! f.m.vanderkleij@utwente.nl fabienne.vanderkleij@cito.nl or