Victoria Crisp & Christine Ward The PePCAA Project: Formative Scenario-based CAA in Psychology for Teachers

1. Victoria Crisp & Christine Ward The PePCAA Project: Formative Scenario-based CAA in Psychology for Teachers

2. Crisp & Ward: The PePCAA project Topics • The PePCAA project and its aims • Scenarios: development, types and CAA features • Examples of scenarios • Results of trialling • Problems in sharing scenarios transnationally • Achievements.

3. Crisp & Ward: The PePCAA project PePCAA – Pedagogical Psychology Computer Assisted Assessment • FIM-NeuesLernen, University of Erlangen • LabSET, University of Liege • Universitat Oberta de Cataluña (OU of Catalonia) • PGU, Göteborg University • UCLES - University of Cambridge Local Examinations Syndicate

4. Crisp & Ward: The PePCAA project PePCAA Scenario-based computer-delivered formative assessments in pedagogical psychology for teachers and trainee teachers. Main aims: • to improve the quality of teaching, by linking psychological theory to practical instances of its use in the classroom • to explore more innovative techniques of computer assisted assessment

5. Crisp & Ward: The PePCAA project Expected benefits of the approach (1): • Intrinsic interest of scenarios • Showing relevance of psychological theory • Increasing trainee awareness of psychological theory and research • Developing awareness of why some strategies are more effective • Stimulating discussion.

6. Crisp & Ward: The PePCAA project Expected benefits of the approach (2): • Allowing trainees to assess own knowledge • Motivational benefits of CAA • Opportunity to experiment with CAA.

7. Crisp & Ward: The PePCAA project Scenario development • Early discussion and drafting of examples • Drafting by one partner • Comment by at least one other partner • Revision and trialling • Development of PLAC (PePCAA Learning Assessment Circle).

8. Crisp & Ward: The PePCAA project Scenario types: • Problem Solving: diagnose causes of a classroom difficulty, suggest possible solutions and their theoretical basis, predict effects of proposed intervention. • Best Practice: plan the best strategy to achieve a goal, justify the strategy by reference to theory, predict the outcomes .

9. Crisp & Ward: The PePCAA project

10. Crisp & Ward: The PePCAA project Metacognition and confidence ratings • Promote awareness of one’s own knowledge • Confidence rating for each automatically scored answer (in some assessments) • Calculation of Confidence, Prudence and Nuance • Required a work-around in Perception.

11. Crisp & Ward: The PePCAA project Some examples of scenario assessments http://www.pepcaa.odl.org/

12. Crisp & Ward: The PePCAA project

13. Crisp & Ward: The PePCAA project

14. Crisp & Ward: The PePCAA project

15. Crisp & Ward: The PePCAA project

16. Crisp & Ward: The PePCAA project

17. Crisp & Ward: The PePCAA project Scenario / software features (1): • a scenario (or ‘case study’) description • option to read a short explanation of relevant psychological ideas first • ‘essay’ question, followed by feedback stating the main points expected • ‘essay’ question followed by a multiple-response question asking which points included, with feedback on choices made

18. Crisp & Ward: The PePCAA project Scenario / software features (2): • range of objective question types • division of some assessments into blocks • ‘branching’ dependent upon the candidate’s response to a question • confidence measurement • explanatory feedback, with weblinks

19. Crisp & Ward: The PePCAA project Trialling in the UK • Small groups of trainee teachers • Scenarios from both UK and non-UK partners • Feedback collected through both open and closed questions • Some follow up in interviews.

20. Crisp & Ward: The PePCAA project Trialling – format of feedback Statement (eg ‘The assessment is easy to understand’) to be rated as follows: +2 strongly agree +1 agree 0 neither agree or disagree -1 disagree -2 strongly disagree

21. Crisp & Ward: The PePCAA project Scenarios trialled in the UK - primary 2: PE lesson with 10 year olds (6 students) 3: Number facts (6) 4: Assessing Science (6) 6: Problem Solving (7) 7: Attachment Theory and Parenting (6) 8: Friendships (6) 9: Play (6) PGU 7 (Sweden): Teaching conceptual change. (6)

22. Crisp & Ward: The PePCAA project Trialling of primary scenarios in UK – Average ratings (scale – 2.0 to + 2.0) Knowledge of pedagogical psychology is very important to me 0.64 My current knowledge of pedagogical psychology is very good - 0.39 Assessment is easy to understand 0.88 Assessment is about a real world situation 1.22

23. Crisp & Ward: The PePCAA project Trialling of primary scenarios in UK – Average ratings (scale – 2.0 to + 2.0) Assessment is relevant for teachers 1.36 Assessment is well structured 0.88 Assessment is too long - 0.33 Assessment is stimulating 0.87 Working on assessment was helpful 0.95 Feedback given was helpful 0.97

24. Crisp & Ward: The PePCAA project Trialling of primary scenarios in UK – Key findings • Generally favourable responses to all scenarios • Ratings for Swedish scenario similar to those for UK • One scenario (UCLES 7) shown to be too long • Amendments made in the light of responses

25. Crisp & Ward: The PePCAA project Scenarios trialled in the UK - secondary 1: History Lesson (7 students) 5: Dance lesson with 11 year olds (7) Labset 2 (Belgium): Why success isn’t explained like failure (7) FIM 8 (Germany): Self-managed learning (7)

26. Crisp & Ward: The PePCAA project Trialling of secondary scenarios in UK – Average ratings (scale – 2.0 to + 2.0) Knowledge of pedagogical psychology is very important to me 0.46 My current knowledge of pedagogical psychology is very good - 0.46 Assessment is easy to understand 0.04 Assessment is about a real world situation 0.54

27. Crisp & Ward: The PePCAA project Trialling of secondary scenarios in UK – Average ratings (scale – 2.0 to + 2.0) Assessment is relevant for teachers 0.54 Assessment is well structured 0.14 Assessment is too long - 0.43 Assessment is stimulating - 0.04 Working on assessment was helpful 0.04 Feedback given was helpful - 0.21

28. Crisp & Ward: The PePCAA project Trialling of secondary scenarios in UK: Key findings • Average ratings much lower than for primary scenarios • Ratings for Belgian and German scenarios much lower than those for UK • Amendments made to UK scenarios in the light of responses.

29. Crisp & Ward: The PePCAA project Trialling of scenarios in UK: Some general findings • Mixed response to open-response questions not automatically marked • Mixed response to multiple-response questions • Improvements required to candidate information and end of test feedback • Some students found the psychological language difficult.

30. Crisp & Ward: The PePCAA project Problems in transnational sharing of scenarios • Mismatch in the coverage of psychology • Depth of treatment and detail differed, even where topics were common • More practical approach in the UK & Sweden • Assumptions about the background to the scenario might not fit another country • Careful editing of language needed.

31. Crisp & Ward: The PePCAA project Achievements of the PePCAA project • A framework for scenario development, plus actual examples • PePCAA Learning Assessment Circle (PLAC) • Showed benefits of scenarios in motivating students and prompting reflection • Demonstrated taking CAA beyond simple multiple-choice. Potential applications outside pedagogical psychology

32. Crisp & Ward: The PePCAA project Try the assessments for yourself on http://www.pepcaa.odl.org/