Race to the Top Assessment Program Consultation

1. Race to the TopAssessment ProgramConsultation Carol Campbell Atlanta, Georgia November 17, 2009

2. Assessment Design Principles Place quality of assessment over quantity Designed around a clear vision of purpose and learning goals Support data- informed decision-making for local and system improvement Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies Are part of a clear and flexible learning trajectory across subjects and over education/career pathways Effective Assessment Systems Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways Encompass and combine assessment of, for and as learning Embody high expectations to engage, motivate, support and stretch all students to progress and achieve Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning Build and develop professional capacity

3. 1. Clear Vision of Purpose and Learning Goals - Specifying 21st century skills, college and career-readiness • Example: High school examinations in England combine: Subject/curricular knowledge with generic skills Functional skills – “essential for life, learning and work” – practical skills in English, mathematics and information and communication technology” (ICT). Example – Math level 1 - “A member of a reprographics team is able to look at statistics and graphs detailing paper usage and identify trends that might suggest that equipment requires servicing” Personal learning and thinking skills (PLTS) – “These are skills that will equip people for successful employment and lifelong learning…PLTS require learners to be: • Independent enquirers • Creative thinkers • Reflective learners • Team workers • Self-managers • Effective participants”

4. Assessing 21st Century Skills – England, Ages 14 - 19 Assessment criteria for Functional Skills – “Assessment must, according the skill areas: • Provide realistic contexts, scenarios and problems • Specify tasks that are relevant to the context • Require application of knowledge, skills and understanding for a purpose • Require problem solving • Assess process skills and the outcome of their application in different contexts Generally assessments externally defined (exam boards) and internally assessed (teachers), including controlled assessments in classroom and projects. Combinations of open and fixed response items varies by subject area and level. Performance assessments increasingly used in range of qualifications – coursework, projects, extended writing Opportunities for accredited work experience in some qualifications, e.g. Diploma composite qualification includes 10 days work experience

5. Ontario, Canada - Achievement Chart – English, Grades 9-12 – Thinking Skills Thinking – The use of critical and creative thinking skills and/or processes

6. 2. Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies • Example: Ontario, Canada • Content Standards • Curriculum for Grades 1-12 for all subjects and courses • Developed through extensive research, benchmarking, and consultation with all stakeholders • All teachers are required to teach the curriculum expectations • Performance Standards (criterion-referenced) • Criteria are described for student achievement of the curriculum expectations for Grades 1-12 • Four levels of achievement • Four categories of knowledge and skills: Knowledge and Understanding; Thinking; Communication; Application • Consistency: • Exemplars of student work for all curriculum • Standard provincial report cards - aligned with content and performance standards

7. 3. Encompass and combine Assessment of, for and as learning Assessment… Of learning - summative assessments to confirm student knowledge, skills, understanding of curriculum, proficiency For learning – formative assessments using information throughout the learning process to investigate student learning, inform instruction and provide feedback to support student progress As learning – assessment as a process of meta-cognition involving student self-assessment and understanding of own learning processes and knowledge

8. Summative and Formative Assessments: Example – The National Assessment System, Scotland, UK FORMATIVE Local authority collection and analysis of information to inform provision and improvement Inspection feedback and subject/quality/improving reports Follow-through inspection activities Formative assessment Personal learning planning Involving learners, and parents And other adults in the learning process INTERNAL EXTERNAL Teachers’ judgements and reports, With local moderation and National Assessments as part of understanding And sharing standards Scottish Survey of Achievement P3, P5, P7, S2 National Qualifications (SQA) International studies HMIE inspections and reports on Authorities and schools SUMMATIVE

9. 4. Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning Multi-faceted assessments used internationally include: • Assessments of knowledge (recall & analysis) and assessments of performance (demonstration of ability to apply knowledge in practice) • Multiple-choice, constructed-response, extended tasks and projects • On-demand and curriculum-embedded elements • Externally-developed (by teachers and developers) and classroom-developed/managed • Scoring made consistent through training, moderation, calibration, and auditing

10. Examples of Assessment Types: Ontario and England

11. Singapore GCE A-Level Examinations

12. Systematic Achievement Reports at the End of Key Stages, England, UK • Key Stage 1 (age 6-7): Classroom evidence is combined with centrally-developed open-ended tests and tasks in English and mathematics, taken when children are ready and marked by teachers, with moderation. • Key Stage 2 (age 8-11): Teachers’ summary judgments of student progress from classroom evidence are combined with open-ended tests in English, mathematics and science; externally marked and results collected nationally. • Key Stage 3 (age 14): External tests were recently abolished. Moderated teacher assessments are used to report achievement levels in all subjects, nationally collected. • No mandatory high school assessments – although most students will participate in GCSEs (age 15-16) and progress to other academic (A/AS level), vocational (diploma) and flexible pathways for progression

13. 5. Embody high expectations to engage, motivate, support and stretch all students to progress and achieve • Example: England, UK: • Making Good Progress Pilot: • Progression Targets to increase the number of pupils who make two levels of progress throughout a key stage • Progression Premium to reward schools which help students who entered a key stage behind national expectations and make good progress

14. The Assessment for Learning Strategy, England, UK Our aims are that: • Every child knows how they are doing, and understands what they need to do to improve and how to get there. They get support they need to be motivated, independent learners on a ambitious trajectory of improvement • Every teacher is equipped to make well-founded judgement about pupils’ attainment, understands the concepts and principles of progression, and knows how to use their assessment judgements to forward plan, particularly for pupils who are not fulfilling their potential • Every school has in place structured and systematic assessment systems for making regular, useful, manageable and accurate assessments of pupils, and for tracking their progress • Every parent and carer knows how their child is doing, what they need to improve, and how they can support the child and their teachers.

15. Applying Knowledge:A Literacy Task, Grade 10 Ontario Task: Write a minimum of three paragraphs expressing an opinion on the topic below. Develop your main idea with supporting details (proof, facts, examples, etc.). • Purpose and Audience: an adult who is interested in your opinion • Length: The lined space provided for your written work indicates the approximate length of the writing expected. Topic: Are today’s famous people good role models for young people?

16. 6. Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways Example: General Certificate of Secondary Education (GCSE) (age 15-16), England • GCSEs are available in over 50 subjects: students choose. • GCSEs normally take 2 years to complete. Many courses offer assessment in each unit rather than only at the end. • GCSEs use constructed response items, and usually include structured classroom-based tasks that are more extended. • Increased emphasis on extended writing and course work. • Controlled assessments are taken under supervised conditions and are either set by the awarding body and marked by teachers or set by teachers and marked by the awarding body. Teachers determine the timing of controlled assessments. Controlled assessments comprise up to 60% of a grade, depending on the subject. • Introduction of assessment of functional skills and personal learning and thinking skills across subjects • Cost of assessing one unit = $10 approx. • Average costs of a GCSE course assessment = $40 approx.

17. ICT Controlled Assessment Task, GCSE England Litchfield Promotions works with over 40 bands and artists to promote their music and put on performances in the UK. The number of bands they have on their books is gradually expanding. Litchfield Promotions needs to be sure that each performance will make enough money to cover all the staffing costs and overheads as well as make a profit. Many people need to be paid: the bands; sound engineers; and, lighting technicians. There is also the cost of hiring the venue. Litchfield Promotions needs to create an ICT solution to ensure that they have all necessary information and that it is kept up to date. Their solution will show income, outgoings and profit. Candidates will need to: • Work with others to plan and carry out research to investigate how similar companies have produced a solution (The company does not necessarily have to work with bands and artists or be a promotions company.) • Clearly record and display your findings • Recommend a solution that will address the requirements of the task • Produce a design brief, incorporating timescales, purpose and target audience

18. ICT Task (cont.) Produce a solution, ensuring that the following are addressed: • it can be modified to be used in a variety of situations • it has a friendly user interface • it is suitable for the target audience • it has been fully tested. You will need to: • Incorporate a range of: software features, macros, modeling, and validation checks (used appropriately). • Obtain user feedback • Identify areas that require improvement, recommending improvement, with justification • Present information as an integrated document • Evaluate your own and others’ work.

19. England: AS and A Levels - Changes to Assessments • A broad range of question types to assess a wide range of skills • Questions requiring extended answers give students the opportunity to demonstrate the full breadth and depth of their understanding • Synoptic assessments that will test students’ understanding of the subject as a whole, including ability to: • Develop a broader and deeper understanding of the connections between the knowledge and understanding set out in the specifications as a whole • Draw together some of the key insights from their studies • Make purposeful use of these in undertaking assessment tasks • Introduction of an extended project as a separate qualification for A level students to add to their study program.

20. 7. Build and Develop Professional Capacity • Example: Teacher collaboration on assessment design and marking, Alberta, Canada • Test design: • Identify student characteristics • Assist in exam blueprint document • Ensure curricular “fit” of the exam • Pilot prototype multiple-choice and written response forms • Help develop writing assignments and their scoring Item building: • Item development sessions held throughout the province involve teachers in the creation of new multiple-choicce and written responses.

21. Teacher Collaboration – Alberta, Canada Reviewing Tests: • Each new examination form is reviewed by a committee that includes classroom teachers. • The committee examines both the written response and multiple choice sections to ensure that the examination is fair, and demonstrates fidelity to the curriculum. Confirming Standards: • Before written responses are marked a committee of teachers meets with examination branch staff to select student work for use in marker training (example papers, training papers and reliability review papers).

22. Teacher Collaboration – Alberta, Canada The Marking Process: • Each January and June approximately 1300 teachers meet to mark the written responses in a process with tight calibration to produce consistency in scoring. Examination Advisory Committee: • Once yearly representatives of various stakeholder groups, including the Alberta Teachers’ Association meet with branch staff to review examination results from the previous school year, and to offer suggestions for the improvement of the examination program.

23. Scoring of Assessments - Ontario Grade 9 Mathematics – raw score points = 46% multiple-choice, 54% open-response Ontario Secondary School Literacy Test – raw score points = 48% multiple-choice, 52% open-response Multiple choice items are machine scored Written open-ended responses systematically scored by qualified scorers (mostly Ontario educators) Use of generic rubric to describe levels of performance, item-specific rubric for each item, and anchors to illustrate descriptors for the score points to the rubric Committee of educators define range of performance within each code of scoring rubric – ‘range finding’ development Every scorer participates in extensive training System of checks and monitoring applied throughout process Quality assurance and statistical techniques to validate interrater reliability and scoring validity Use of equating process to place student scores on common scale over two adjacent years and to determine cut scores

24. Using Assessments to Inform Improvements: Example: Ontario

25. 8. Are part of a clear and flexible learning trajectory across subjects and over education/career pathways • Example: Exam boards in England ensure consistency of standards within subjects while allowing flexibility of approach across subjects. • Five exam boards - governed by specifications, criteria, frameworks and oversight of Office of the Qualifications and Examinations (OFQUAL) Regulator • Exam boards: • set out the course syllabus & specifications to meet criteria set by the regulator • oversee writing of exam questions and how exam papers should be marked • check coursework/controlled assessments • provide training and quality control for examiners, including use of ‘standardization meetings’ to compare marking across examiners • set grade boundaries to align marks and standards

26. National Qualifications Framework, England:Learner Pathways

27. 9. Support data-informed decision-making for local and system improvement

31. MiSA GiARE • Funding • Policy/Programs • Public Reporting Ministry of Education Ontario School Information System (OnSIS) EQAO Board Validation & Sign-off by Director Encryption & Depersonalization Data Staging Area Ministry Validation Elementary Secondary Data Warehouse (ESDW) Statistics Canada Data Other Sources Province-Wide Information Data Student Management Systems (SMS) School Boards/Authorities Data Data Data Data repository or data warehouse Decision Support System Info Info Info EQAO Students Teachers, Principals & Administrators Board Administrators Stats Can Other

32. 10. Place quality of assessment over quantity Examples: • Australia – National standards (underway) State Syllabi, State and Local Assessments (3 grades plus high school) • Canada – Provincial standards, syllabi & assessments (2 grades plus high school) • England – National standards & curriculum School-based assessments + national tasks (1 grade plus high school) Five secondary examination boards • Singapore – National standards and curriculum National exams + school-based tasks (1 grade plus high school)

33. Common Practices Across Countries Assessments are part of a tightly integrated system of standards, curriculum, instruction, assessment, and teacher development at the state or national (in small countries) level Assessments include evidence of actual student performance on challenging tasks that evaluate 21st century standards of learning Teachers are integrally involved in the development and scoring of assessments Assessments are designed to continuously improve teaching and learning Use of multiple-choice testing limited and combined with other forms of assessment Emphasis on criterion-referenced and curriculum-embedded assessments Not same approach for every student, every grade, every year Increasing use of teacher assessments in national reporting, which external summative assessments in specific years/transition points

34. Principles Proposals • Start with the end in mind – clarify and confirm vision for US high school graduates as engaged citizens, active contributors within the global economy and future leaders. Requires combination of functional skills, generic and specialist knowledge, performance competencies, and personal learning strategies • Integrate assessment design with common standards for consistency of curriculum expectations and clarity of teaching and learning standards. Involves criterion-referenced and curriculum-embedded assessment items. Attention to consistency within curricular areas. Vertical alignment over grade levels to support progression of learning. • Designed around a clear vision of purpose and learning goals • Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies

35. Principles Proposals • Develop an assessment system focused on learning where summative assessments are one element of a wider repertoire of assessment information. Increase use and consistency of teachers’ assessments in reporting and monitoring student achievement and progress. • Use range of item-types and assessment approaches, including use of performance assessments, extended writing and project work. Use, but limit, time on standardized multiple-choice items and test taking. • Encompass and combine assessment of, for and as learning • Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning

36. Embody high expectations to engage, motivate, support and stretch all students to progress and achieve Build and develop professional capacity Principles Proposals • Range of assessments, including items and tasks that involve authentic experience, critical thinking and applied performance of students. Attention to student personal learning plans and progression targets. Growth and cohort analysis. Develop capacity for student self- and peer-assessment, including group work. • Educator capacity building is critical and requires time, resources (human, financial, material) and support. Teacher moderation is a powerful strategy to support teaching and learning throughout school year. Assessment items, constructs, interval data and other technical advice are required to be available and accessible to identify linkages between assessments, curriculum expectations, instructional strategies and interventions for specific students.

37. Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways Are part of a clear and flexible learning trajectory across subjects and over education/career pathways Principles Proposals • Implement repertoire of formative and summative assessments with variety of approaches within and across grade levels to demonstrate knowledge, understanding and application. • Support clarity of learning pathways through high school and into college and career readiness, with accreditation options and credit accumulation requirements to indicate when a student is on or off track for success

38. Support data-informed decision-making for local and system improvement Place quality of assessment over quantity Principles Proposals • Implement data infrastructure (technology and systems) and culture (human capacity and data access) to enable collection, analysis and use of data at all levels of system • Shift emphasis from multiple-choice standardized tests as main approach for every student, every grade, every year. Combine external tests with coursework and other approaches to assess learning over time and in applied contexts. Develop consistency across assessment aligned with common standards, quality assurance mechanisms, and attention to building consistency in educator understanding and application of assessment strategies, including teacher moderation.