The Power of Assessment: Creating Valid and Fair Tests

1. SQA Qualifications for the Future:Masterclass SeriesPurposeful AssessmentProfessor Gordon StobartInstitute of Education - London

2. The power of assessment Unspeakably more depends on what things are called than on what they are…creating new names and assessments and apparent truths is enough to create new ‘things’. (Friedrich Nietzsche, 1887) The individual in contemporary society is not so much described by tests as constructed by them (Allan Hanson, 1996) I’m no good at spelling ..and I’m hopeless at times tables so I’m frightened I’ll do the SATS and I’ll be a nothing. (Hannah age 11, Reay & Wiliam 1996)

3. Ruth the Pragmatist • Learning the formula for each exam and practising it endlessly. I got an A1 in English because I knew exactly what was required in each question. I learned off the sample answers provided by the examiners and I knew how much information was required and in what format in every section of the paper. That’s how you do well in these examinations…There’s no point in knowing about stuff that is not going to come up in the exams. I was always frustrated by teachers who would say ‘You don’t need to know this for the exams but I’ll tell you anyway’. I wanted my A1 – what’s the point of learning material that won’t come up in the exams?

4. Keeping validity central • Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. Validity is, therefore, the most fundamental consideration in developing and evaluating tests. (AERA/APA/NCME Standards 1999, p.9)

5. Construct validity • What is the construct (target domain) that an assessment is sampling? • How is the construct/domain/curriculum determined and by whom? • Messick’s two threats to validity: • Construct underrepresentation • Construct irrelevant variance • Examples: English; readability; rote learning • Fairness is part of validity

6. Principles for valid test development • Make explicit the purpose and learning demand: • Aims, rather than content, should determine the purpose and form of the assessment; • Learning demand – what kind of understanding? The role of test format;

7. Principles for valid test development 2. Encourage ‘principled’ knowledge through less predictable questions: • Move from ‘when you…’ to ‘what if..’; • Classroom do not have to always mimic external assessments (rich questions; misunderstandings)

8. Principles for valid test development 3. Keep it as authentic as possible • Hanson’s ‘fabricating quality of tests’ • Dependability – optimising construct validity and reliability (and manageability) 4. Make them fair • Whose knowledge and skills? • Reflective test development – the importance of openness; • Assessment cocktails