Building a Comprehensive Assessment System Edward Roeber

1. Building a Comprehensive Assessment System Edward Roeber Office of Educational Assessment and Accountability 1

2. This is the time of assessment…. • This is the era of standards-based reform, based on student assessment information • Federal laws require much assessment at the state level • District response … more assessment! • Using assessment information is now “high stakes” - school survival now depends on it • Yet, educators are not always prepared to develop assessments nor use the information provided by them 2

3. Added Assessments • New assessments will add even more external pressures • Michigan Merit Examination • End-of-course examinations • With added state level assessment, as well as district-level assessments, our assessment systems may be out of balance - too much external testing, too little classroom-based assessment 3

4. What’s Needed • We need a balanced assessment system - one that honors the work in classroom assessment and is aligned to the state and district assessments • A variety of approaches to assessment are needed - perhaps more assessment, but of different kinds • All assessments needs to work in the same direction - a coordinated system of assessment 4

5. What is a Balanced Assessment System? • Formative • Supports on-going learning in the classroom • Interim • Provides periodic snapshots of student achievement at the district level • Summative • Assesses student learning at the conclusion of a phase of learning • All parts work together and are equally important 5

6. Balanced Assessment Systems • Are all elements of the assessment system coordinated? Are all three types present? • Formative • Interim • Summative • Are the content and skills measured compatible? • Are standards, curriculum, instruction, learning, assessment and professional development aligned 6

7. Balanced Assessment Systems • Is assessment continuous - do the assessments measure student progress over time (days, weeks, months, school years, and school careers)? • Are a variety of formal and less formal assessment methods used on an on-going basis? • Is assessment integrated - does the assessment system fit in the larger educational system? • Is assessment seamlessly integrated into instruction? 7

8. Balanced Assessment Systems • Are a variety of high quality assessments being used? • Do all assessments meet professional standards (reliability, validity, and so forth)? • Are the assessment purposes clearly spelled out and are assessments used that can achieve these purposes? • Do all students participate in assessments appropriate for them? • Do assessments have positive consequences - for students, educators, parents, and others? 8

9. What are Summative Assessments? • State Level • Michigan Educational Assessment Program • Michigan Merit Examination • MI-Access • English Language Proficiency Assessment • Secondary Credit Assessments? • District Level 9

10. Uses of Summative Assessments • District • Prioritize schools for improvement assistance efforts • Provide data for the local school board to understand student and school achievement • Report academic progress of the district to parents and other taxpayers, including key community decision-makers • Increase community support for schools and for school improvement efforts • Key Question: Did students learn what they should have? 10

11. Uses of Summative Assessment • State • Determine priority schools for the interventions required by Federal (and state) law • Guide state-level assistance to “high priority” schools - those continuing not to make adequate yearly progress • Fund schools - could be general state aid and/or special programs to impact struggling schools • Allocate human resources - such as providing staff or consultants to provide supportive management advice and strategies to schools 11

12. What are Interim Assessments? • Periodic assessments given school- or district- wide at fixed times during the school year • Measure the school’s or district’s curricula in important content areas • Provide evidence that all students have been taught key skills • Serve to reinforce a common curriculum and sets of learning experiences 12

13. Interim Assessments • District-constructed assessments, or • Commercially-available assessments • Replicas of the MEAP assessments • Pacing assessments that follow the curriculum • Assessments that do not follow MEAP nor the district curriculum • Key Question: Are students in each school on track for proficiency? 13

14. Interim Assessments • Examples • Quarterly tests based on the instructional sequence(s) used to measure the pacing guides used instructionally • Advantages • May determine if students are learning the enabling skills • Might catch students who are experiencing difficulties in learning before they fall far behind • Challenges • Instructional sequences are not always universal - are multiple forms of the assessments needed? • Are the assessments of high quality and/or match instruction? 14

15. Uses of Interim Assessment • Increase instructional alignment - horizontal and vertical • Inform parents of all students about the level of student achievement and improvement efforts under way • Assure consistency of instruction across schools and the district • Program evaluation - evaluate the effectiveness of new initiatives 15

16. What is Formative Assessment? • Classroom-based assessments used on an on-going basis in every classroom • Carefully-thought-out strategies to engage students in learning in and outside of the classroom • May encourage different ways of learning - moving from passive to active student learning • May encourage student self-assessment/self-monitoring 16

17. Formative Assessment • Teachers are engaged actively in the assessment process by: • Sharing instructional targets with students • Questioning students • Observing students • Examining and evaluating student work • Providing feedback to students • Determining instructional implications of the assessments • Conferencing with students, other teachers and parents 17

18. Formative Assessments • Students are actively engaged in the assessment process by: • Learning what is expected of them • Taking responsibility for their own learning • Actively participating in the learning process • Participating in how students document what they have learned and how they learned it • Demonstrating their learning to peers, educators and their parents 18

19. Uses of Formative Assessment • Guide student learning on a daily basis by providing information about what critical skills were and were not learned • Provide extra learning opportunities to students who are struggling academically • Provide additional learning opportunities for students who are doing well academically • Report student progress to students, parents, and other educators • Key Question: Has each student learned? 19

20. Characteristics of Both Large-Scale and Classroom Assessments • Shared model of student learning • Should include learning progressions over time • Shared concept of disciplinary knowledge and competence • Focus assessment on the most valued knowledge and skills • Focus on understanding and reasoning, not recall • Assess enabling skills and procedural knowledge • Signal to teachers and students what is most important • Base assessment on clearly written standards 20

21. Characteristics of Both Large-Scale and Classroom Assessments • Measure a manageable body of knowledge and a limited number of the most important skills • Target general forms of cognition • problem solving and inductive reasoning • skills that are more domain-specific • Move from single-answer assessments towards greater use of tasks • content based • measure rich and well-structured knowledge • are open to multiple approaches 21

22. Characteristics of Both Large-Scale and Classroom Assessments • Designed to be valid and useful to support decisions (large-scale and classroom) • Are technically sound and timely • Are designed in accordance with the purpose for which results will be used 22

23. Characteristics of Both Large-Scale and Classroom Assessments • Measure the knowledge and skills they purport to measure • Are designed to be reliable, valid, and fair for the inferences that will be drawn from results • Report results in enough detail to reveal needed instructional changes and highlight deficiencies in the system or in instruction • Focus on student learning in school, rather than what students have learned outside of school 23

24. Characteristics of Both Large-Scale and Classroom Assessments • Provide opportunities for students of different backgrounds to connect their knowledge to relevant school expectations. • A range of measurement approaches are used to provide a variety and range of evidence of student achievement • Provide multiple opportunities for students to demonstrate competence • Look at differences in how students perform in different assessments as rich data about students, not “error” 24

25. Unique Characteristics of Large-Scale Assessment • Provide comparative data (normative or standards-based) • judge the adequacy of performance • determine specific curricular/instructional areas for improvement • Provide quality feedback to educators about patterns of errors that could be target for instruction in the future 25

26. Unique Characteristics of Large-Scale Assessment • Must be cost-effective and feasible • instructional uses must be worth the time taken from instruction • Results must be reported to stakeholders 26

27. Unique Characteristics of Classroom Assessments • Must be on-going and integrated seamlessly into instruction so teachers and students receive frequent feedback on their progress • Assess some desired proficiencies that cannot be effectively assessed on large-scale assessments • Provide quality ongoing feedback to teachers about patterns of errors that could indicate the need for modification of future instructional strategies 27

28. Unique Characteristics of Classroom Assessments • Help teachers to identify and reconstruct students’ misconceptions • Provide quality feedback to students about their performance and specific guidance about how to improve • Help students learn how to assess their current levels of understanding in relation to learning goals; learn to recognize and produce quality work 28

29. What are ways we can assess students? • Selected-response tests • Forced-choice - multiple-choice, true-false, matching • Written response - short-answer, extended response • Observation • Structured • Unstructured • Samples of Student Work 29

30. What are ways we can assess students? • Performance Events • Individual interviews • Hands-on performance assessments • Performance Tasks • Individual assessments • Group performance assessments • Projects • Datafolios • Portfolios 30

31. Forced-Choice Items • Examples • multiple-choice, true-false, matching • Advantages • Can cover much content in little time • Inexpensive • Easy to score - “Objective” • Challenges • Difficult to write high-quality items • May stress memorization over understanding and application • Could encourage instruction that emphasizes content coverage over deep understanding of concepts 31

32. Written Response Items • Examples • essays, short-answer, solve math problems, fill in the blank • Advantages • Tap student understanding • Can measure application of knowledge • Can be administered in group settings • Challenges • More time-consuming, expensive to score • Scoring may not be objective • Students may not respond 32

33. Structured Observation • Examples • A topic for group discussion question by small groups of students • Teamwork exercise with observation protocol • Science laboratory exercise • Advantages • Assess application of skills • Can observe students using skills in real contexts • Challenges • Difficult to observe multiple groups (or individuals) at the same time • Time consuming; labor intensive 33

34. Unstructured Observation • Examples • Student classroom participation • Quality of student responses to teacher’s questions • Advantages • Naturalistic observation of students • Collect data (systematically) not usually collected or used • Challenges • Lack of opportunity for all students to be observed • Anecdotal in nature - how generalizable? 34

35. Samples of Student Work • Examples • Daily classroom work • Term papers • Advantages • Captures work already being done in the classroom • Could reinforce the on-going creation of products by students • Challenges • The information from each student may be different • May be difficult to “standardize” across multiple students 35

36. Performance Events • Examples - • Musical performance sing, play a musical instrument • Science experiment • Solve mathematical problem • Advantages • Often, these assess the most important skills • Challenges • Time consuming to assess all student • Expensive to develop, use, and score 36

37. Performance Tasks • Examples • Paper on an English topic • Compose a song • Plan for a school or community improvement project • Advantages • Sustain and extend classroom learning • Encourage students to work deeply in the content area • Challenges • Time consuming - for students and teachers • Students need support and assistance 37

38. Projects • Examples • Senior project on a topic of the student’s interest • Implementation of a plan to improve the school or community • Advantages • Students can engage actual work that interests them • Challenges • Very time consuming for both students and teachers, to carry out, to present, and to score 38

39. Datafolios • Examples • Collection of data from one or more sources, such as observation, ratings of student work, test scores in one folio • Advantages • Captures and organizes information about a student in one place • Challenges • Time consuming for students and/or teachers to select and organize the information • Data itself may not tell much of a story; narrative commentary may also be needed 39

40. Portfolios • Examples • Showcase of best arts products • Collection of essays written in English class • Advantages • Show student progress • Capture best work of the student • Encourage students to produce work • Challenges • Very time consuming to develop, collect, organize, and score • May be idiosyncratic to the individual students 40

41. Questions to Consider First • Do we know what students need to learn? • Do we understand what students need to learn? • How do students best learn? • What instructional strategies are most effective? • How do we know that students have learned? • Can state, district, and classroom assessments work together to promote quality student learning? 41

42. Questions to Consider First • What work on assessment is needed to improve how assessment contributes to student learning? • What professional learning needs to occur for this work to happen? • How can the state encourage work on all three parts of the balanced assessment system? 42

43. Choosing Assessment Methods • Each educator and school should use a variety of assessment methods (multiple methods) • The methods selected locally (district, school, and classroom) should complement and build on the assessments used at the state and national level • Coordinated assessments are most useful, since they can present a more complete picture of student achievement and other accomplishments 43

44. Coordinated Assessment Systems • Use both large-scale and classroom-based assessments • Measure the same skills or related skills using different measures of related skills • One possible model: • State or district: end-skills • Classroom: learning progressions leading to the accomplishment of end skills 44

45. Coordinated Assessment Systems • Developed to provide “comprehensive” information on learning and achievement • Different assessment methods may or may not yield consistent information • If the results are consistent, still use multiple methods • If not consistent, look for reasons - may yield insights into student behavior and/or the quality of the measures used. • Avoid using multiple assessments as a multiple-choice item - selecting one source and ignoring the others 45

46. Developing the Needed System • Examine the academic content standards valued at the state level • Format - end-skills or learning progressions? • Breadth and depth - how thorough? • Completeness - few or many in number? • Determine how the state will assess these • What types of measures will be used? • What types of measures are missing? • What other types of data do classroom teachers need? 46

47. Developing the Needed System • If the skills being measured are end-skills, then create the desired learning progressions in grade-level expectations • Work should involve educators from all levels to assure horizontal and vertical alignment • Determine the assessments methods to be used to assess the learning progressions • Determine assessment methods needed to gauge student learning • Develop the needed assessments 47

48. Developing the Needed System • Create and implement instructional plans to provide the instruction on learning progressions • Make sure all teachers are “on the same page” • Horizontal alignment • Vertical alignment • Use assessments to determine student learning • Embed assessments in on-going instruction • Review and use assessment results to improve instruction • Examine assessment results to determine individual student and group instructional needs 48

49. Michigan’s Assessment System • In Michigan, the assessment system needs to be more fully balanced • Summative assessments play a major role • Interim assessments are needed • Formative assessments are also needed • Teachers need to be given time to actively discuss instructional strategies • Assessments need to be embedded in good instruction • Teachers need to be formally prepared to work on the assessments to be used 49

50. Secondary Credit Assessment Program • State and districts need to determine whether students have learned enough to receive credit in a variety of high school credit areas • This could be done in conventional end-of-course exams or through a different manner • We propose to do both - to develop a balanced assessment system in Michigan • We invite interested districts, schools, and educators to become partners in this effort 50