The Role of Information in Systems for Learning

1. The Role of Information in Systems for Learning Paul Nichols Charles DePascale The Center for Assessment

2. Problem • Recognition that improved student learning requires coordinated assessment, instruction, curricula and professional development • States and districts are being asked to conceptualize and implement assessments as a component of broad and coherent systems • For example, comprehensive assessment systems or systems for learning. • New burdens on district and state staff

3. Problem • Think beyond a single test or set of tests to coordinated system of assessment, curriculum, pedagogy and professional development • Manage and held accountable for the award, development and implementation of multi-component systems • Cross department boundaries, engage multiple-vendors and involve many times the cost of a testing program

4. Status Quo • State and district staff already have or can find the following training and tools to help with test development and implementation for individual tests • Currently no theoretical framework and no indices to help state, district and other educational leaders in planning, developing and evaluating systems for learning

5. Systems Approach • Move beyond hand waving to offer a framework and quantitative tools used by district and state staff to conceptualize and implement systems for learning (SFL) • Note the components that probably should be included in an SFL • Address SFL coherence by looking at communication of information from students to teachers • Present analytic framework allowing district and state staff to predict SFL coherence across different configurations

6. Defining Systems for Learning • What is a system? From the field of systems thinking: A system is “a collection of parts which interact with each other to function as a whole” • An SFL is a set of components which interact with each other to function as a whole with the intention of improving student learning

7. Defining Systems for Learning • Paul Black and colleagues describe the components of an SFL:

8. Defining Systems for Learning • Curriculum consists of the knowledge and skills in subject areas that teachers teach and students learn. • Instruction refers to methods of teaching and the learning activities used to help students master the content and objectives specified by a curriculum. • Assessment is the means used to measure the outcomes of education and the achievement of students with regard to important competencies. • Theory of learning is the model of student cognition and learning in the domain

9. Defining Systems for Learning • Karin Hess reminds us that professional development is an important missing component:

10. Defining Systems for Learning • Coherence is provided by the theory of learning • The components must “talk to each other” in a common language • The language of communication is the learning theory

11. Measuring SFL Coherence • An SFL may be conceptualized as a communication system • Think of a radio or your iPad receiving a transmission • A generic communication system

12. Measuring SFL Coherence • Signal is generated by an information source • Transmitter codes the signal and passes the coded signal to the receiver • Receiver decodes the signal and passes the reconstructed signal to the destination • Noise is likely to degrade the information in the signal as it is passed from component to component • Information is contaminated by noise so that information received is different from an error-free message

13. Measuring SFL Coherence • Teacher attempting to understand the information relevant to the construct in student performance

14. Measuring SFL Coherence • From Measurement Theory to Information Theory • Information is defined as the amount of uncertainty reduced • Student whose answer is right or wrong vs student whose answer reveals stage of learning progression • Bit is measure of amount of information in terms of choice or uncertainty • Learning outcome between two equally probable alternatives is learning one bit of information

15. Measuring SFL Coherence • Coherence can be defined as the degree to which different components help communication • H(x|y), equivocation, is uncertainty that x was transmitted when y is received • H(x|y) = • Mapping from x to y reduces noise or error the value approaches 0 • Lower values = higher coherence

16. Measuring SFL Coherence • Components hinder/facilitate communication so decrease/increase coherence • Performance-based assessment where the teacher is looking for formative information • Scaffolding reading or constructed response on the circulatory system • Learning progression with 5 levels • Focus on professional development & instruction

17. Measuring SFL Coherence • Simulation 1: Professional development addresses basic statistics, reliability, validity • Fails to address learning progressions • Teachers’ scoring shows no relationship to students’ learning progression stage • Students receive instruction unrelated to prior understanding • Coherence low, H(x|y) = 2.32

18. Measuring SFL Coherence • Simulation 2: Professional development addresses the concept of learning progressions and describes stages in the learning progression • Teachers’ scoring shows large “halo effect” and noise • Students with impoverished understanding fail to receive instruction • Coherence moderate, H(x|y) = 1.49

19. Measuring SFL Coherence • Simulation 3: Professional development addresses concept of learning progressions, describes stages and in the learning progression, calibrates scoring with exemplars • Teachers’ scoring shows some “halo effect” and impact of noise • Most students receive appropriate instruction • H(x|y) = 0.83

20. Conclusion • State and district staff are asked to conceptualize and implement SFLs • State and district staff need a theoretical framework and quantitative indices to help plan and evaluate SFLs • Measurement theory focuses on individual tests or groups of tests • Instructional theory focuses on instructional techniques and activities • The framework and index of system coherence are a start…