Dr Barbara Kelly, Carole Edgerton, Elaine Robertson, Diann Neil

Educational Psychology in Scotland Annual Conference Herriot Watt UniversitySeptember 14th 2012The Preschool PATHS Curriculum:Using Implementation Science to Increase Effectiveness Dr Barbara Kelly, Carole Edgerton, Elaine Robertson, Diann Neil Glasgow South Psychological Services

Outline of Presentation • What is PATHS? • Preschool PATHS project in Glasgow South • Focus on Implementation Science • Interim Audit of IMPLEMENTATION • Glasgow PATHS Network Building Sustainable Programmes • Fuller discussion and references for all the issues raised can be found in Barbara Kelly and Danny Perkins 2012, Cambridge Handbook of Implementation Science for Psychology in Education

PATHS Curriculum in Glasgow • Fairer Scotland Preschool PATHS project one of few externally funded in Glasgow`s education context • PATHS primary curriculum is becoming well known throughout Scotland and has a developing evidence base locally helping to fulfil Glasgow`s vision for children's well being and welfare • Preschool PATHS is a fairly recent development in Glasgow and is currently being embedded in a number of preschool establishments • Preschool project focuses on Implementation Science with a view to increasing sustainability and maximising effectiveness

Why choose PATHS and Preschool PATHS? • Recognised as one of only eleven `blueprint` programmes recommended for reducing adolescent crime, aggression and substance abuse in the US (Mihalic et al 2001) • Significant positive outcomes for ASN pupils and in mental health contexts ( Greenberg et al, 2000) • Study in England demonstrated greater cooperation, empathy and self control (Curtis and Norgate, 2007) • In Scotland enhanced pupils` ability to manage difficult emotion and positive impact on behaviour ( Kelly et al, 2004) • Key paper for Preschool PATHS effectiveness Domitrovich et al 2009 Reflects government priorities for developing children`s mental health and well being

Why choose PATHS? • Evidence of impact in primary and preschool PATHS is building globally • Underlying dynamic, theoretical framework proving to be more and more relevant to our understanding of child development and learning • The central concepts of integration of functioning and the importance of self- regulation have anticipated research findings and are key emerging theoretical models in developmental psychology

PATHS Impacts • Offers a range of strategies to recognise manage and regulate feelings and develop alternatives to hostility and violence • Impacts on attainments –literacy and language • Linked closely to cognitive skills and problem solving: emotional understanding underpins effective thinking But emotional education may have wider effects: literature from theory of mind research and developmental psychology suggests that emotion content of conversations between parents and children creates feeling, thinking and behavioural templates

PATHS in ACTION 1

Focus on Implementation Science The way a programme is implemented is more important than the programme itself • Implementation is an emerging Science- Journal of Implementation Science appeared in 2006 • Barbara Kelly and Danny Perkins (2012),Cambridge Handbook of Implementation Science for Psychology in Education, Cambridge University Press • Offers conceptual and applied frameworks and strategies for effective implementation

Basic Implementation Strategies (Blasé et al 2012) Who, why and what in building effectiveness • Recruitment and selection: choose those who are interested in the programme and keen to be involved. • Training: provide in depth pre and in service training on both theory and practice issues linked to the programme • On going Consultation and Coaching: follow through initial training with coaching support in context for practitioners. • Monitor and evaluate Staff performance: monitor how staff use skills to target input at all stages • Inform decision making and policy review: provide data to support on-going local investment decisions • Facilitative administrative support: have clear leadership and support to implement effectively and sustain programmes effectively • System Support: Ensure the availability of financial, organisational and other resources- don’t run on a shoe string

Design of the project Every new implementation of any programme presents a new ecology and different strengths and weaknesses • Two year implementation process • This reports on first year audit • Used Audit Methodology - support and check the progress made in implementing evidence based programmes and allows feedback and changes to improve effectiveness • Monitors successes and obstacles along the way • Poor implementation results in low impact

So what have we actually done? • Following the Implementation Drivers identifies by Blase et al 2012 we focused on key areas to support implementation: First -Set up a steering group involving Principal Psychologist, Senior EP, EPs, Quality Improvement Officer, Early Years Co-ordinator to guide the implementation processes Recruitment and initial training processes for staff • Presentation to volunteer nursery head teachers • Completion & return of the Readiness Checklist • Selection of nurseries • 4 nurseries with 10 children in each as the in-depth research group

Assessing Readiness Aarons et al 2012 • Readiness for evidence based practice is a concept developed by Aarons • Arose from medical context where practitioners were unable or unwilling to implement evidence based programmes successfully • Reasons are linked to processes outlined earlier: values , attitudes, understanding, resources, leadership………and `diamond hard resistance`

PATHS readiness checklist • Asked volunteers to consider a range of issues which they would have to address to implement PATHS effectively • This was used to select candidates for the study • Not an infallible method as reality always proves more challenging but alerts practitioners to implementation demands

Extensive data collection throughout and at end of year one Impact on children • Kusche Emotional Inventory (Domitrovich et al) • The British Picture Vocabulary Scale 3 • Demographic Questionnaire • Preschool Behaviour Checklist • Number of children involved was based on the statistical analysis of the Domitrovich study – 18 children required to demonstrate effects. • Within Subjects Design to evaluate the impact of Pre-school PATHS This design makes the assumption that we know Pre-school PATHS works and looks at its effects on individual children in different early years contexts.

Implementation, training and support activities • Major innovation in Glasgow South -Coaching and Modelling by Peripatetic Preschool Team • Inset to all nursery staff on Pre-school PATHS & Implementation Issues • Signed agreement to follow implementation guidelines • Discussion with staff about frequency and quality of lessons (30 required for significance) • Observation of some lessons for fidelity and engagement lessons • Interview with staff about impact after year one

Statistical Analysis after one year • Repeated measures ANCOVA looked at scores in Kushe Emotional Inventory and the British Picture Vocabulary Scale (BPVS). • The co-variants controlled were: gender, age, number of lessons taught, English as an Additional Language (EAL), ethnicity and Additional Support Needs (ASN). • Firstly the nursery results were looked at together as a whole population (n=30). • Only thirty of the 40 target children had a full data set

Emerging positive trends • The results of the ANCOVA for the Kushe Emotional Inventoryshowed no significant overall effect of time at this stage; F(1,23)=0.881, p=0.358. • However, the mean percentage of correct answers on both did increase from time 1 (mean= 59.83%) to time 2 (mean=63.33%).

Checking by establishment • T- Test significant on both the BPVS and Kushe in two establishments but low numbers of children hampers confident interpretation • Backward trends began to emerge in two establishments

Interim and End of Year One Outcomes Staff views via telephone interviews • Overall feedback is very positive. • All note positive changes in children's emotional and social development and behaviour. • All felt PATHS achieved its theoretical objectives very well. • Respondents emphasised key processes of children`s identification and empathy with puppet characters and the value of scenarios played out by characters which generalised to real situations with teacher prompts • All will continue into year two and report developing confidence in delivery and management of the lessons

Little data to date on the early stages of implementation-the installation and initial implementation phases (Schaffer, Nesselrodt and Stringfield (1997) • Many programmes fail during the installation phase because of lack of investment of time and resources • Leadership role is crucial in maintaining a focus on quality implementation and challenging current values, beliefs and paradigms which do not fit the new way of working • No one has a crystal ball and this phase is characterised by rapid problem solving cycles which respond to issues emerging in the audit process

Why were results not reaching significance? Audit of early implementation stages Training-not all received follow up training Number of lessons- no establishment carried out the planned thirty lessons Coaching- varied in style, frequency and experience: some staff began doing lessons themselves when the agreement had been on modelling in the early stages, some did not find the coach helpful.

Implementation and child results • These are reassuringly in a strong positive direction in two establishments • In the other two establishments problems arose probably impacting on child results Staff continuity, knowledge and experience of coaches, knowledge and training of staff, frequency of lessons, resource issues and confidence, fidelity of lessons taught.

Staff reactions to Implementation Issues • Majority nurseries felt that the in-service presentation and taster sessions fired imagination • Coaches too slow to get off the ground re lessons • Some difficulties with language barriers – high numbers of EAL parents and children in one nursery • Two nurseries reported feeling unsupported initially and that the coach had poor skills • Resource and time issues re organisation and photocopying • Difficulties across job share staff and communication with them • Extension activities were too difficult.

Reaction of parents • Nurseries reported that parents seemed to be interested and enthusiastic but they hadn’t set out to involve them • One set up a PATHS display and this had stimulated a lot of conversation • One reported that parents had come and asked to see the materials • Staff feel that parent involvement would be helpful in the implementation phases

What’s gone well? • Most reported coach had been excellent and built confidence in staff to deliver on their own • Programme is really clear and very comprehensive • Materials are excellent and children are really enjoying the programme • Two nurseries report evidence of children discussing PATHS at home with parents • Three feel impact is clear already

Next phase • Each establishment to complete 30 lessons before child assessment • Iron out problems via close collaboration between nursery EP and staff • Carry out a final audit after two years or thirty lessons and when staff are seen to deliver lessons with fidelity and confidence • Established PATHS Network to foster good implementation of PATHS and other programmes, promote self evaluation and to develop links to global PATHS networks

Implementation Science: The Practitioner is the intervention • Offers a sound evidence base and guidelines for building accountable evidence based practice across all aspects and levels of Psychological Service Delivery • Success for children ultimately depends on staff readiness to deliver high quality evidence based teaching, care and programmes with enthusiasm, depth of understanding of theory and skill. • Supporting those processes is a major part of our work but is demanding and intricate.

Dr Barbara Kelly, Carole Edgerton, Elaine Robertson, Diann Neil