Dr Andrea Baldwin Creative Industries Faculty QUT

1. Evaluating Dr Andrea Baldwin Creative Industries Faculty QUT a Train the Trainer program in Papua New Guinea

2. WHAT’S THE PROJECT? • Sexual health – focus on STIs and HIV • Behaviour change communication using applied theatre • Experiential learning, not one-way dissemination of health information • Train the trainer model • Ultimate beneficiaries are young people (15-25) • Developing hybrid forms of applied theatre for education/ health promotion

3. WHO ARE THE TEAM? • Professor Brad Haseman Creative Industries, QUT • Associate Professor Anne Hickling Hudson Education, QUT • Dr Andrea Baldwin Creative Industries, QUT • Ms Hayley Linthwaite Creative Industries, QUT • Ms Jane Awi • Creative Industries, QUT • Mr Martin Tonny • Research and Administrative • Assistant • Ms Jackie Kauli • Creative Industries, QUT • Research Advisory Groups • and Participants in Field Sites

4. WHO ARE THE PARTNERS? • Australian Research Council – Linkage Grant • Queensland University of Technology • Griffith University • National AIDS Council Secretariat PNG • University of Goroka • University of Papua New Guinea

5. Evaluation Framework Haseman, 2006 Stufflebeam, 2003

6. THE MAP IS NOT THE TERRITORY

7. Spiral Out: • who learnt what from this project? • what impacts did the project have – positive and negative, expected and unexpected? • what does this mean for the future? • descriptive, expansive answers • qualitative, performative • Spiral In: • what’s the bottom line? • did implementation go to plan? • did the project create the desired change? • what are the take-home learnings? • yes/no, definitive answers • quantitative

8. RESEARCH/EVALUATION METHODOLOGIES MUST: • Be culturally appropriate, meaningful and relevant to the participants and other stakeholders • Be sensitive to issues of advantage/ disadvantage for individuals and groups of stakeholders • Complement and advance the aims and approach of the overall project

9. A CONCEPTUAL METAPHOR PROJECT What have we learnt about propagation? PROGRAM Is the seed viable?

10. EVALUATION RESEARCH • Involves determining the worth, merit, or quality of an evaluation object, such as an educational program • Formative evaluation – helps in the design, implementation and improvement of the program • Summative evaluation – helps make decisions about whether a program should be supported into the future Johnson & Christensen, 2008

11. MAGIC PILLS? • Drug trial model • Laboratory • Quantitative, use of inferential statistics • Clear distinctions between the agent (drug) and conditions of administration (dosage, interactions, etc.)

12. PARTICIPATORY ACTION RESEARCH • Much more organic – can’t make such clear distinctions between the agent (program, action, education materials) and the context (community or organisation) • Requires more holistic approach to collecting and analysing data, interpreting findings, making recommendations for future

13. IMPORTANT TO: • Capture negative/undesirable impacts of the project as well as positive/desirable ones • Capture and assess unexpected impacts of the project as well as intended/expected ones

14. NEW PHASE, NEW APPROACH The Life Drama Program The Train the Trainer Program Life Drama program content established Train the Trainer program content and structure established – Handbook and Bilum Collect qualitative and performative data from two sites • Development highly participatory – Tari, Madang, Karkar Island • Quantitative, qualitative and performative data collected • Triangulating and synthesising vast quantity of data from participants and other stakeholders

15. WHAT MORE HAVE WE LEARNT ABOUT THE SEED? • Have the educators themselves acquired new knowledge and attitudes in relation to sexual health and HIV? • Interview/focus group data • Performative data 2. Do the educators believe their students will acquire/have acquired new knowledge and attitudes through the training? • Post and follow-up interview data

16. WHAT MORE HAVE WE LEARNT ABOUT PROPAGATION? • What factors enhance or impede • the delivery and effectiveness of training Life Drama trainers? • the formation and functioning of a network of trainers?

17. 2. What are the similarities and differences between training community educators and teacher educators, which are likely to impact differently on the effectiveness of these two groups as trainers? 3. What are the similarities and differences that are likely to impact on their ability to support one another as a functioning network?

18. GENERALISABILITY? • These questions being investigated in PNG – two specific sites/groups – capture realistic picture of this specific scenario • Interpretation of data will include the attempt to draw out general principles that might apply in other settings

19. “BASIC” EVALUATION

20. PROGRESS REPORTS – is the project being implemented as designed? Is the research team doing what it said it would do? Are budget milestones and timeframes being met? If not, why not? (emphasis on Context, Inputs and Processes) • FINAL REPORT– did the project achieve intended outcomes hoped for? What factors enhanced or hindered the achievement of these outcomes? (Emphasis on Context and Products)

21. OBJECTIVE 1: TRAINEE GROUP • To train between 20 and 40 Life Drama trainers, with an equal balance of male and female trainers, through two training hubs: • University of Papua New Guinea, Port Moresby, National Capital District • University of Goroka, Goroka, Eastern Highlands

22. EVALUATION Quantitative data – how many males and how many females completed training? Performative data – how many trainees passed requirements for certification?

23. REQUIREMENTS FOR CERTIFICATION By completion of the training, trainees will demonstrate - • A practical knowledge of contextual, communicative practices and their place in society; • A deeper awareness of community issues, in particular sexually transmitted illnesses and HIV; • Specific skills in using drama techniques confidently to help community groups explore community issues, in particular sexually transmitted illnesses and HIV; • Leadership skills which help community groups find their own solutions to those problems and barriers which are preventing widespread understanding of, and positive response to, the challenges of sexually transmitted illnesses and HIV; • Critical, analytical and creative approaches about connections between purposes, audiences and diverse ways of communication; • Individual and group interaction leading to brain-storming, understanding and action; • Attitudes appropriate to inclusive communication practices for cultural, linguistic, gender differences and collaborative contexts.

24. OBJECTIVE 2: TRAINER NETWORK 2. To support the trainee group to function as a “trainer network” – sharing practice, exchanging ideas, assisting one another’s professional development, and expanding knowledge of the Life Drama program beyond the original group of trainees

25. EVALUATION Qualitative data – post-training and follow-up interview and focus group data on factors that may enhance or hinder/have enhanced or hindered the functioning of the trainer network

26. OBJECTIVE 3: TRAINER EFFECTIVENESS 3. To assist participants to integrate Life Drama techniques with their existing educational skills and strategies, to enhance their effectiveness as trainers of HIV-related material

27. EVALUATION Qualitative data – post-training and follow-up interview and focus group data on how trainees intend to use/ are using their Life Drama skills for HIV education purposes Performative data – review in-training video footage of trainers’ practice, and follow-up practice

28. OBJECTIVE 4: RESEARCH CAPACITY To build research capacity in Papua New Guinea: • Participatory Action Research project involving senior staff in arts health areas of University of Papua New Guinea and University of Goroka • Develop research skills of PNG national members of the research team • Encourage research participants to gather data as “co-researchers”

29. EVALUATION Quantitative data – how many PNG national staff from the two universities are actively involved in the project? Qualitative data – what is the level of engagement/mutual benefit for the PNG national staff and the QUT research team? Qualitative data – Quality of reflective journalling and reporting by participants as co-researchers

30. GROUP 1 UNIVERSITY OF PAPUA NEW GUINEA PORT MORESBY

31. Port Moresby organisations: • University of Papua New Guinea • Lecturing staff • HIV peer educators • AnglicareStopAIDS • National Capital District Commission • Centre for Democratic Institutions PNG • Family Health International PNG • Motukoita Community Youth Development • Department of Education COMMUNITY EDUCATORS Hela Province Organisations: Porgera Joint Venture PNG Police Service Hela Rural Women’s Foundation OilSearch

32. GROUP 2 UNIVERSITY OF GOROKA GOROKA, EASTERN HIGHLANDS

33. TEACHER EDUCATORS • University of Goroka: • U100 (mandatory HIV course) • Language and Literature Department • Social Sciences Department • Expressive Arts Department • Science Faculty • Education Faculty • Early Childhood Department • Student Services • Madang Teachers College • Oxfam PNG • Independent Community Theatre Practitioners

34. OBJECTIVE 1: TRAINEE GROUP Port Moresby • 6 females • 14 males • 20 TOTAL • Goroka • 15 females • 11 males • 26 TOTAL TOTAL = 46 TRAINEES, 25 MALE AND 21 FEMALE

35. OBJECTIVE 2: TRAINER NETWORK • Only post-training focus group data available so far (follow-up focus group data to be collected in November 2011) • Only in-training video performative data available so far (follow-up video of trainees’ practice to be collected in November 2011)

36. Factors Enhancing • Membership of same organisation (eg. university staff) • Ability to access communication technology (internet, email, phone) • Organisational support (esp. U of G) • Ongoing relationship with QUT • Ongoing resourcing

37. Factors Impeding • Membership of different organisations • Working with different client groups – different needs • Communication barriers (few trainees have internet access; internet access often unreliable in PNG; phone access ditto; phone communication expensive) • Lack of resourcing

38. OBJECTIVE 3: TRAINER EFFECTIVENESS Qualitative Data: • Only post-training focus group data available to date. Follow-up focus group/ interview data will be collected in November 2011

39. Performative Data: • Only in-training video available for review to date. Follow-up video will be collected in November 2011 • Video to be analysed by members of team expert in: • Drama-in-Education/ Theatre for Development (Creative Industries Faculty) • Pedagogy (Education Faculty)

40. OBJECTIVE 4: RESEARCH CAPACITY • # PNG senior university staff actively engaged in project, not as trainees: 7 (5 in Goroka, 2 in Moresby) • “Active engagement” evidenced by: participating in training sessions, observing training sessions, providing additional supports and assistance to the program, contributing to focus groups

41. Level of mutual benefit for PNG national research staff: • 2 PNG national research staff undertaking PhDs with QUT as a result of Life Drama project (assisting with data collection and analysis for Train the Trainer program) • 1 PNG national research staff member now undertaking Masters with another university, partly as a result of involvement in Life Drama • QUT and University of Goroka now exploring future partnership opportunities

42. Quality of reflective journalling and report provision by participants – yet to be assessed (November 2011)

43. TWO GROUPS: COMPARE AND CONTRAST

44. CONTEXT Port Moresby Goroka Primarily teacher educators Primarily local (living and working in Goroka) Most trainees affiliated with University of Goroka at some level • Primarily community educators • Significant percentage from outside Moresby or away from the university (eg. settlement areas) • Large range of organisations represented, sometimes by 1 or 2 people

45. INPUTS Port Moresby Goroka Two indoor venues in good repair but not always available; using outdoor spaces without disturbing exams More organisational commitment – more resourcing – more condusive learning environment Issues over resourcing for non-local attendees • Less organisational commitment – less resourcing – more difficult learning environment • Indoor venue in poor repair • Several trainees self-funding attendance – resentments over resourcing

46. PROCESSES Port Moresby Goroka Handbook available throughout training, used for daily review of lessons Trainees all comfortable with English • Handbook not available until last day • Trainees generally less comfortable with English

47. PRODUCTS Port Moresby Goroka Easier to follow up and evaluate We expect more effectiveness of trainer network because most participants have access to communication mechanisms and infrastructure, and/or are geographically co-located • Harder to follow up and evaluate • We expect less effectiveness of trainer network because fewer participants have good access to communication mechanisms and infrastructure

48. WHAT MORE HAVE WE LEARNT ABOUT THE SEED? • Have the educators themselves acquired new knowledge and attitudes in relation to sexual health and HIV? • Interview/focus group data – yes, eg. difference between HIV and AIDS, how to live healthy life with HIV • Performative data – yes, eg. condom demonstrations

49. QUALITATIVE DATA “I thought AIDS was an instant disease. But I realise now there’s a harmony in the body that tries to fight against this disease. If I’m thinking that, what about the students? We need to teach them all the things you can do to stay healthy” Female participant, early 50s, lecturer in U100 HIV program at Goroka University. Post-training focus group.

50. PERFORMATIVE DATA