against the odds…. the challenge of assessing design innovation

1. against the odds…. the challenge of assessing design innovation Richard Kimbell Technology Education Research Unit Goldsmiths: University of London Feb 2005

2. DfES d&t steering group design & technology its distinctive contribution in the curriculum • “Design and technology prepares pupils • ...to intervene creatively to improve the quality of life • ...to become autonomous and creative problem solvers • ...as individuals and members of a team • ...to combine practical skills with an understanding of aesthetics, social and environmental issues and industrial practices. • …to reflect on and evaluate present and past design and technology, its uses and effects. • Through design and technology, all pupils can become discriminating and informed users, and can become innovators.” • (QCA / DfEE 2000)

3. the problem of assessment lies in the portfolio problem/brief investigation/research ideas design solution making testing

4. as a result, the portfolio is • (typically) not creative - but formulaic • (typically) not done in teams - but individually • (typically) not innovative - but safe

5. research design

6. the first assessment framework 'wow''yawn' risky innovative formulaic predictable havingideas (for starting points) growingideas (towards a solution) • modelling to explore & develop ideas • optimising & synthesising growth provingideas (to confirm effectiveness)

8. “idea objects” to provoke design thoughts

9. modelling resources to support ideation

10. a group work table - for 3 students

11. the start … with 21 students

12. teacher introduces the modelling resources

13. modelling enhancing ideation

14. the development process in action

15. the photo story-line • launched as a means of recording - to support assessment • it became a powerful developmental force - students were prepared to butcher and adapt existing models - safe in the knowledge that they had a good record of it - students were looking forward to the next one - delighting in the progress they were making

16. idea modelling • alongside sketching • no sequence • no priority concept modelling development modelling proving modelling

17. “post-its” kick-start the 2nd morning

18. final reflections

19. “your name in lights” “jack-in-a-box light”

20. having ideas growing through modelling holistic growing through synthesising proving through criticality

21. teacher comments…. • Students enjoyed the challenge. Great atmosphere in the room. Students were totally engaged for the majority of the time. Loved the photographs! • Students were developing products that were more interesting and creative than those that materialise from the brief, specification, research, ideas, development route! • Through a highly focused activity, students had the opportunity to experiment with a concept that may or may not work. Given permission to take a risk and not to be too worried about a quality outcome •  One of the remarks that I recall from reading the self assessment sheet was " it shows what I can do in a positive way" – this was written by a pupil who is a school phobic and finds school work difficult. • Able to use their imagination (have a free hand) • Students realising how much they can do in a short time

22. students’ reactions... • Hull • AB112 - Had other people to give their opinions on your ideas • AB114 - Helping my group with different ideas • AB119 - Being able to use my own ideas how I wanted to • AB1111 - These have been the best two days in school • AB1117 - Its easier to understand when you try it out for yourself • Staffordshire • AB226 - Good layout of worksheets • AB228 - You got to show your ability • AB2210 - Starting from scratch to make you think • Wales • AB314 - Using your own ideas ; making models instead of drawing - they work better • AB315 - It was fun to design something and actually make it • AB319 - Seeing your idea develop • AB3110 - Only having a limited time to work • AB3111- We got to try out different things, using our ideas • AB323 - Being able to make something by myself and what I wanted to make • AB324 - I realised that I could do more than I thought I could • AB326 - The photos was the best way to show the steps, which I think was good

23. after the 1st task ….. … a series of tasks • 8 in total • 2 with each GCSE exam board (WJEC, Edexcel, AQA, OCR) • across the design & technology spectrum • product design • systems & control • graphics / product • textiles • food the 6 hr activity structure was held constant for all these tasks: • early ideas • team development • photo story-line • red pen • team reflections • fast forwarding etc etc etc

24. Context & task effect performance variation across 2 tasks • in 23% of cases the marks are identical • in a further 30% of cases there is a difference of 1 mark • in a further 26% of cases there is a difference of 2 marks • in a further 9% of cases there is a difference of 3 marks • in a further 5% of cases there is a difference of 4 marks • in a further 6% of cases there is a difference of 5 marks In 80% of cases, the scores on two tests varied by 2 marks or less on a 12 point scale Variability is higher in some schools than others

25. other interesting findings…. • the most innovative students are not identified by GCSE predicted grades 2 of the 4 most innovative students were predicted ‘C’ • the most innovative students are not predicted by ‘general ability’ measures the high scoring group in KS3 SATs contained very low innovation levels the low scoring group in KS3 SATs contained very high innovation levels • girls (still) generally outperform boys the higher the ‘ability’ group, the more they outperform boys the lower the ‘ability’ group the less they outperform boys ie. girls performance has more spread than boys performance

26. issue 1 : autonomy & constraint procedures interventions team ideas learners’ expressions and outcomes becomes more detailed and richer photo storyline learners’ thoughts and ideas become more sophisticated red pen team reflections post-it valuing ideas blue pen fast forward • “Being able to use my own ideas how • I wanted to” • “You can make things you didn't expect” • “I liked getting myimagination go wild” • “Seeing your idea develop” • “Being able to make something by myself • and what I wanted to make” each procedural intervention pushes ideas forward but without prescribing what those ideas should be or in what form they should be represented.

27. issue 2 : the power of modelling • ‘Faking the future’ describes the rough and ready IDEO formula of building lots of crude prototypes....Kelley describes this as ‘fast fearless prototyping’ (Myerson 2001) “Using your own ideas ; making models instead of drawing - they work better” “Its easier to understand when you try it out for yourself” “You can make things you didn't expect” ”Its fun designing- using different materials”

28. against the odds…. the next challenge of assessing design innovation project ‘e-scape’