1 / 22

Panel session 2: All in the process

Building Evidence in Education: Conference for EEF Evaluators 11 th July: Theory 12 th July: Practice www.educationendowmentfoundation.org.uk. Panel session 2: All in the process. Pleasure and pain of process July 2013. Introductions and purpose. Becky Clarkson

trula
Download Presentation

Panel session 2: All in the process

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building Evidence in Education:Conference for EEF Evaluators11th July: Theory12th July: Practicewww.educationendowmentfoundation.org.uk

  2. Panel session 2: All in the process

  3. Pleasure and pain of process July 2013

  4. Introductions and purpose Becky Clarkson Look at some of the issues I’ve encountered with process evaluations, and hopefully some of the positive aspects.

  5. Overview of the five transition RCTs

  6. What the process evaluations consist of • Observations of training sessions • Observations of intervention sessions • Interviewing deliverers • Teacher log

  7. The first pain TIME

  8. The second pain (the biggy really!) • RELATIONSHIPS • non-contactable • too interested • over-friendly

  9. The pleasures Personal interest Continuity

  10. Questions or comments? My contact details: r.clarkson@nfer.ac.uk

  11. Some Issues in Process Evaluation Presentation to the EEF evaluators conference Caroline Sharp July 2013

  12. What is a process evaluation? Process evaluation: documents and analyses the development and implementation of a programme, assessing whether strategies were implemented as planned and whether expected output was actually produced. Bureau of Justice Assistance (1997) quoted in EEF (2013)

  13. What does process evaluation involve? It: • explores the implementation and/or development of a programme • considers how the programme has been implemented and/or developed • can assess whether the programme has been implemented and/or developed as planned • considers which activities/outputs have been implemented.

  14. Why implementation is important Interventions are rarely implemented as designed and, crucially, variability in implementation is related to variability in the achievement of expected outcomes (Lendrum and Humphrey, 2012, p.635). Because we need to know why and how an initiative has achieved its outcomes and impact in order to draw valid conclusions and improve practice.

  15. Some issues and challenges • Process evaluation may be confused with qualitative evaluation • It can be difficult to establish intended activities and outputs • How to respond to new/developing initiatives? • Difficult to achieve good measures of resources and costs • How to deal with issues of fidelity versus responsiveness to context? • How to make good judgements about scaleability?

  16. What helps? • A clearly defined intervention (rather than a loosely defined ‘programme’) • A more established programme model • A clear logic model or Theory of Change • Participation in key events (e.g. observing staff training; attending project meetings) • Good communication between the programme managers and the evaluation team(s) • Understanding the broader context of going to scale.

  17. Process evaluation of Changing Mindsets: challenges and lessons

  18. Process evaluation of Changing Mindsets: challenges and lessons • Changing Mindsets • Primary school intervention (Year 5) • Managed and delivered by Portsmouth University and partners • In Portsmouth and Hampshire • Testing theory of ‘fixed’ and ‘growth’ mindsets • 2 separate interventions: Pupil; Teacher Inset • RCT with tests in English, Maths, and Mindset measures

  19. Process evaluation of Changing Mindsets: challenges and lessons • Process evaluation – general objectives • To establish whether the programme was delivered as planned • To identify: • factors which could explain why the intervention did/didn’t work • contextual reasons for variation • Issues which should inform plans for future roll-out

  20. Process evaluation of Changing Mindsets: challenges and lessons • Design of the process evaluation • Include experiences and perspectives of participants (project workers, teachers, head teachers, project partners) • Cover all key components of the intervention • Keep to the ‘light touch’ EEF principle • Use project’s own evaluation

  21. Process evaluation of Changing Mindsets: challenges and lessons Process evaluation of Changing Mindsets – challenges Not over-burdening the project or participants Gaining cooperation without jeopardising project Avoiding confusion between project team and evaluator team Fitting in with the project: timing and place Keeping up with changes to the project Staying objective

  22. Process evaluation of Changing Mindsets: challenges and lessons • Process evaluation: some lessons from Changing Mindsets • Work with the project team • Don’t duplicate data collection • Use project’s own evaluation • Keep it ‘light’ – don’t over-collect • targeted visits & observations • keep interviews short & focused • Flexible methods - phone rather than face to face, email responses

More Related