SQA Seminar Glasgow, 8 April 2009

1. SQA SeminarGlasgow, 8 April 2009 Sweden’s experiences in curriculum and assessment development Gudrun Erickson University of Gothenburg, Sweden Department of Education

2. Outline • The Swedish school system • Curriculum: current system – discussions – (proposed) changes • Assessment: current system – discussions – (proposed) changes • Collaborative approaches to national assessment • National and international experiences • Concluding remarks – Common concerns • DISCUSSION

3. The Swedish school systemhttp://www.skolverket.se/sb/d/190 • Highly decentralized; main responsibilities at local level • National level (e.g.): Education Act; Curricula; Syllabi for subjects, including grading criteria; National tests • A goal and criterion referenced system • Preschool; 9-year compulsory school; upper secondary education; adult adult education; universities… (http://skolnet.skolverket.se/polopoly/utbsys-eng/) • > 98% of students continue to three-year upper secondary education • Development dialogues; individual development plans; written reports; formal grading (3/4 point scale), from school year 8 • Teachers responsible for grading • Extensive system of advisory national testing and assessment

4. The Swedish National Agency for Education Advisory council Director-general´s office Director-generalDeputy director general Information Legal secretariat Information service Internal audit Education National evaluation National development Administration Quality development. Human resources Preschool and Compulsory school Education statistics School improvement Finance Upper secondary school Result analysis Government funding ICT Adult education Analysis and research Assessment and testing

5. Syllabi and local planning National level Why this subject? What direction? How? (Very limited guidance) When? (Limited guidance) Local level Whythis to reach goals? What(exact) subject matter? How will we work? When to teach what?

6. Balancing act Flexibility Quality assurance Local independence Equity / Equality

7. Discussions & (Proposed) Changes •Increasing clarity of curricula and syllabi, including indications of essential content areas •Increased quality control •More grade levels (from 3/4 to 6 + “no basis for grading”) •Earlier start of formal grading (from school year 8 down to 6) •Earlier national tests; national tests in a wider range of subjects •Individual written documentation from school year one Ongoing revision of national curricula and syllabi

8. Aims of the current national testing/assessment system •Enhance individual educational achievement; •Clarify curricular subject goals and indicate clearly strengths and weaknesses in individual learner profiles; •Concretize curricular subject goals and grading criteria; •Enhance equity and fairness in assessment and grading; •Provide statistics for local and national analyses of educational achievement Advisory function

9. The national assessment systemshould not: • Determine the precise choice of content and methods To be done in collaboration between teachers and students • Function as examinations Teachers to award grades based on the assessment of students’ accumulated work; Consequently, the national tests have an advisory function; however, to what extent is not defined

10. National assessment materials • Formative materials / Diagnostic materials, including models for self and peer assessment A widening range of subjects • National tests: Summative subject tests (mandatory) Extensive materials, including guidelines An increasing number of ‘core’ subjects (Sw, Maths, Eng, Science…) • Electronic assessment bank(formative & summative materials) A widening range of subjects The Swedish National Agency for Education commissions different university departments in the country to take responsibility for test development (and research)

11. Reactions to national tests • Students usually positive to varied, authentic, “different” tasks/tests • > 95 % of teachers positive to the national tests function, content, level of difficulty, guidelines, individual proficiency profiles, layout… • [Teacher] discussions, e.g., about workload; weight/importance of test results; adaptation/accommodation; standards… • [Political] discussions, e.g., about number of tests; aim(s) of national tests; role of national test; stability over time; reliability…

12. Test grade: national subject tests (2008)year 9 – upper secondary education %“Not pass”/ Pass Pass with Pass with Fail distinction special distinction Swedish 93% 41% 45% 11% B11% 46% 34% 9% Mathematics 917% 48% 26% 10% A23% 45% 23% 9% English 94% 35% 43% 18% A6% 39% 43% 11%

13. Final Grade & national Test Grade– end of compulsory school (year 9) – 2008 FG<TG FG=TG FG>TG Swedish6 % 79 % 15 % Mathematics2 % 74 % 24 % English7 % 84 % 9 %

14. Can teachers’ ratings be trusted? • Joint study (2008) by the Swedish NAE and four university departments responsible for test development; to be published shortly on the NAE website • Random sampling:100 teacher-rated grade 9 tests of Swedish, Mathematics and English (oral subtests not included, since recording is not a requirement) • Re-rating by three independent raters; 10-point scale • To some extent, varying results between subjects (higher correlations for Mathematics and English than for Swedish) • Example: English Listening & Reading comprehension (50/50 constructed and selected response): r = > .99; Written production (“essay”): r = .86-.93; Generalizability coefficient: .85[Analyses of oral test results in similar studies show comparable results]

15. Collaborative approaches to national assessment • Political level (NA) – Universities • University level (national and international) • Test development (researchers from different disciplines, teacher trainers, teachers, students…) • Local test administration • Rating by teachers • Reporting by teachers (results and reactions) • Data collection –Analyses – Research • Public reporting /Dissemination

16. A collaborative test development process (FL) • Analyses of literature, research, curricula, examples of tests • Development of items and tasks based on common, explicit principles and specifications • Piloting with a limited number of students > Adjustment of tasks • Large scale pre-testing in randomly selected classes in the country (n ≈ 400 students/task); systematic collection of feedback from all participating teachers and students • Analyses of test results and feedback (a "q+q” approach) • Selection/Sequencing of tasks; Standard setting and Benchmarking • Nation-wide test administration • Collection of data /Analyses / Research > Reporting (publicly available on the web)

17. Contributors and contributions • Teachers and Teacher trainers Development groups; Task/Item construction; Mini trials – Pre-testing (administrating, observing, analysing, discussing, reporting): Selection, composition, sequencing, standard setting; Rating and benchmarking; Reporting and responding after administration of tests… • Students of ‘all’ ages Providing information and sharing their views in interviews and (regular) questionnaires in connection with pre-testing; comments on, e.g, relevance; perceived level of difficulty; content; vocabulary; clarity of instructions; time, speed of speech; Retrospective, task related self-assessment… • Researchers Aspects of language; Gender; Quantitative properties of item and test data; Teachers’ handling of constructed response; Progression in tasks and tests; Dimensionality; Rater introspection; Test-taker and teacher feedback…

18. Current discussions • Number/Type of aims fornational assessment system? •Clarifying/Strengthening the role of national tests? •Formative and Diagnostic materials – where, how and from/by whom? •Effects and implementation of computer assisted testing and assessment? Aspects of validity? Linear – Sequential – CAT..? •Increasing standardization - if yes, concerning what, and how? •Clearer emphasis on aspects of reliability (equity)? •How to avoid narrowing the curriculum - negative impact/washback? •Emphasizing common basic principles for all types of assessment !

19. Good [language] testing and assessmentaccording to approximately 1 400 teenagers in 10 European countries Breadth and Variation Focus on ”usefulness” / communication Learning potential Clarity Fairness Enough time Challenge (Erickson & Gustafsson, 2005) www.ealta.eu.org/resources.htm

20. EALTAwww.ealta.eu.org European Association for Language Testing and Assessment • Broad membership Teachers, Teacher trainers, Developers of large scale testing systems • Non-commercial • Individual membership free • Founded with support from the EU • Resources page; Discussion list for members Annual conferences (June 2009: Turku, Finland) 797 individual members (41 European countries); 81 associate members (27 countries, 5 continents); 51 institutional members; 16 expert members

21. EALTA Guidelines for Good Practice in Language Testing and Assessment Adopted in 2006; Currently translated into 34 languages Translations freely available on the web Address the activities of the three membership categories via questions Initial emphasis on basic principles foralltypes of assessment: TRANSPARENCY Respect for students/examinees, responsibility, fairness, reliability, validity, collaboration among the parties involved

22. Concluding remarks – Common concerns • Creating a reasonable balance between local and central initiative, interpretation and responsibility • Maintaining high standards of validity as well as reliability • Aiming for positive impact on learning and teaching • Creating a reasonable balance, and maintaining a close cooperation, between R & D – research and development / researchers and test developers • Further developing and elaborating methods of collaboration with wide groups of stakeholders • Bridging the gap between formative and summative assessment – emphasizing common, basic principles