RPCV 113 Chris Wagner’s M&E and Recommendations

1. RPCV 113 Chris Wagner’sM&E and Recommendations “Therefore, not only do we need to call upon our business partners to help us identify what success will look like, but we will need a cooperative effort throughout the learning and performance process.”

2. What I will cover today • The Kirkpatrick Four Levels of Monitoring and Evaluation: Reaction, Learning, Behavior and Results • The Kirkpatrick’s Five Foundation Principles • Analysis of Appreciate Inquiry in Phrae, Peace Corps volunteer questionnaire feedback, and informal feedback. • And how we can use this feedback in developing Peace Corps Thailand’s M&E systems; including general recommendations. • Conclusion: Creating outcomes based on the “real life” experience of Peace Corps volunteers, counterparts, co-teachers, supervisors, and field supervisors at site.

3. The 4 Levels of Monitoring and Evaluation • Level 1: Reaction • To what degree do the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors react favorably to the learning/training events: PST1, PST2, Mid-Service Training, COS and other training events.

4. The 4 Levels of Monitoring and Evaluation • Level 2: Learning • To what degree do the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors acquiretheintendedknowledge, skills, and attitudes based on their participation in the learning/training events such as PST1, PST2, Mid-Service Training, COS and other training events.

5. The 4 Levels of Monitoring and Evaluation • Level 3: Behavior • To what degree do the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors apply what they have learned during trainings such as PST1, PST2, Mid-Service Training, COS and other training events when they return to site.

6. The 4 Levels of Monitoring and Evaluation • Level 4: Results/ Targeted Outcomes • To what degree do the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors reach the TARGETED OUTCOMESthat occur as a result of the learning events after trainings such as PST1, PST2, Mid-Service Training, COS and other training events when they return to site. AND how does thePeace Corps Thailand staff conduct the subsequent required reinforcement.

7. Conclusion: The Easy Part • Most of us understand the first two steps • Step 1: Reaction—We want to design a training event that the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors will REACT favorably to. • Step 2: Learning—We also know, or we should always be learning, what the knowledge, skills, and attitudes we want the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors to LEARN DURING THE TRAINING EVENT. • Side Note: I will come to this point later, but more work needs to done by the Peace Corps staff to have a better idea of the challenges the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors face at site.

8. Conclusion: HOWEVEVER, the Hard Part • It is more difficult to monitor Step 3 Behavior and Step 4 Results: • Step 3: Behavior— How do we know what degree the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors apply what they learned during training when they are back at site? • Step 4: Results-- How do we know what degree the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors reach the targeted outcomes? And how does the Peace Corps staff subsequently reinforce what is taught during PST1, PST2, Mid-Service Training, and other trainings once the training is over?

9. The Kirkpatrick Method: The Five Foundation Principles • Foundation 1: The End Is The Beginning (we start at level 4) • Most organizations only reach Steps 1 and 2 (Reaction and Learning) when they train their employees, or in our case, the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors at events such as PST1, PST2, Mid-Service Training, and other trainings. • When an organization can also reach Steps 3 and 4 (Behavior and Results) then the Peace Corps staff knows that the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors are applying at site what they are being taught during PST1, PST2, Mid-Service Training, and other trainings! • The BIG QUESTION the PC Thailand staff must answer is how do we monitor what is being taught at PST1, PST2, Mid-Service Training and other trainings.

10. The Kirkpatrick Method • Foundation 1 Continued: The End is the Beginning • Step 1: Peace Corps Thailand staff must identify what is essential for the volunteer to know at site in order to do their work at site. What are our DESIRED RESULTS/OUTCOMES/INDICATORS? • Step 2: Peace Corps Thailand staff must DETERMINE the attitudes, knowledge, skills that will bring about the desired results. • Step 3: The final challenge for the Peace Corps Thailand staff is to present the training program in a way that enables the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors volunteers to be successful in reaching the desired results. • This is a continuing process that should be improved each time a PST 1 or PST2 for example is planned– thus volunteer feedback, along with feedback from counterparts, co-teachers, supervisors, and field supervisors is essential.

11. The Kirkpatrick Method • Foundation 2: RETURN ON EXPECTATIONS • Too often trainings likes PST1, PST2, Mid-Service Training, COS and other trainings are based on the past experiences of the PC Staff in Bangkok and/or previous Pre-Service Trainings– while these experiences are an important part of planning the next training, more must be done. • Thus, for the Peace Corps Staff to gain an accurate picture of what volunteers need in the field, PC Staff must be more involved in understanding the challenges volunteers face at site, and then create clear desired outcomes when designing a PST1, PST2, Mid-Service Training, or other training events.

12. The Kirkpatrick Method • Foundation 2 Continued • Thus, PC Thailand should design its PST1, PST2, Mid-Service Training, COS, and other training events by IDENTIFIFYING the desired TARGETED OUTCOMES of the training based on data from the field which can only come from the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors . . . including observation from PC Staff in the field! • This should be the main driving force in designing any PC Thailand training. • We should not be afraid to change our approach to any Pre-Service, Mid-Service, COS or other training, IF the changes are based on what we learn from the field.

13. The Kirkpatrick Method • Foundation 2 Continued: Return on Expectations • Peace Corps staff must engage the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisorsin order to more fully understand “What success will look like” at the volunteers’ site. • These indicators of success then become your Level 4 Results or Desired Outcomes which can the help you more sharply define what is needed during a training. • By doing this, we become better at meeting our Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors’ expectations.

14. The Kirkpatrick Method. Foundation 3: PC Staff Partnership is necessary to bring about DESIRED OUTCOMES • The above diagram is from research by Bersin and Associates 2008. The diagram shows us that 70% of employee/volunteer learning takes place on-the-job in many different ways. THUS, we/PC STAFF need to reinvent our roles by becoming experts in and deeply involved with learning that extends beyond our normal comfort level in Bangkok, or we run the risk of not providing effective training to the volunteers.

15. The Kirkpatrick Method Foundation 3: PC Staff Partnership is Vital Continued • Therefore, not only does the PC Staff need to use the PC volunteers, the co-teachers, the counterparts, and the field supervisors to help PC Thailand identify what success will look like, BUT we need a cooperative effort throughout the learning and performance process between the Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors. • Peace Corps staff needs to be more involved in the field, shadowing volunteers, making more phone calls, identifying key informants such as exceptional volunteers, counterparts, field supervisors, and/or co-teachers who can assist the PC Thailand Staff in continuing to develop more effect training programs year after year.

16. Foundation 4: Value must be created before it can be demonstrated • Second Study: “Typical organizations invests 85% of its resources in the training event, yet those events only contributed to 24% to the learning effectiveness of the participants. Typical Learning Environment

17. Step 4: Value in Reinforcement • Second Study Continued . . . The activities that lead to the most learning effectiveness were follow-up activities that occurred after the training event.”

18. Step 4: Value in Reinforcement continued • What does this mean? We are putting most of our time into designing, developing, and delivering training (Levels 1 and 2) and only getting about 25% of the benefit. We are spending none of our time on the follow-up activities that translate into positive behavior change and results (Levels 3 and 4) that we intend our training program to deliver.

19. Kirkpatrick Method Foundation 4: Value Must Be Created Before It Can Be Demonstrated. • Before PST1, PST2, Mid-Service Training, COS and other trainings PC Staff must prepare the participants for the training. • PC Staff must explain to Peace Corps volunteers, co-teachers, counterparts, supervisors and field supervisors what they will learn in training. • Why it is important • Expectations before and after training event • And the kind of support they will receive throughout the process, including, once the training is over.

20. Foundation 4 Continued: PC Staff Partnership is Vital • “Even more critical is the role of the PC Staff after the training. You are the key people to reinforce newly learned knowledge and skills through support and accountability. The degree to which this reinforcement and coaching happens directly correlates to improved performance and positive outcomes.

21. Step 4: Value in Reinforcement • One Study: “Even providing excellent training does not lead to significant transfer of learning to behavior and subsequent results without a good deal of deliberate and consistent reinforcement.

22. Step 4: Value in Reinforcement continued • One study demonstrated that 70% of training failure comes after training is completed! • Major Factors: • Employees/PC volunteers didn’t get a chance to apply what they learned soon enough after training. • A training program that did not follow up and non-existent coaching. Causes of “Training Failure”

23. Kirkpatrick Method • Foundation 5: A Chain of Evidence • When we focus on levels 1 & 2 we are basically shouting “Look how much this is costing you!” • HOWEVER, if we work backwards, starting with Level 4 to Level 3 to Level 2 and then to Level 1 we are shouting “Look how much value we are bringing!”

24. Kirkpatrick Method • Foundation 5: Chain of Evidence cont… • Level 1 and Level 2 show how much time and how many resources have been invested in training. • Level 3 and Level 4 show the value and tangible results that training has delivered.

25. Peace Corps Staff’s Bottom Line • “It is essential that learning professionals (the Peace Corps Staff) redefine their roles and extend their: • Expertise, Involvement, Influence, Impact in the field in order to better support the Peace Corps volunteers, their counterparts, supervisors, co-teachers, and field supervisors! • In addition, such efforts will assist the counterparts, co-teachers, supervisors, field supervisors in understanding the role of the volunteer, and how best to support the volunteer.

26. Thoughts on Training • Before each and every training session be it PST1, PST2, Mid-Service Training, COS or other training event we need to do the following: • Design the training based on a targeted outcome that has proven to be of use to volunteers/counterparts in the field • Explain what the following training will cover • Explain why we are doing the training at this point in time (sequencing) • How the volunteer/counterpart will be able to use/apply the training when they reach site • Be open to all questions and if you don’t have the answer then as a staff you need to find the answer and follow up with the PC volunteer/counterpart etc who asked the question. • Explain how you will follow up to see how they are doing…or always remind them that the training staff at PC HQ is always here to answer questions/help them through the process. • Always have some kind of evaluation of the training event after each and every session, build in the time. We can only improve ourselves as teachers/trainers if we know what our strengths and weaknesses are.

27. OK, Here we are • The following few slides include feedback from the Appreciate Inquiry Activity I did in Phrae, informal discussions with volunteers, and the PST 121 Combined Feedback form. • The question I want all of us to think about is this: • How many of these problems could be solved by learning more about the volunteer’s work at site, and then developing clear outcomes for us to reach during PST1, PST2, Mid-Service Training and other trainings?

28. Phrae, PST1 & PST2 Feedback • As we go through the slides I want you to consider the volunteer’s point of view during training, and the strange new world they find themselves in once they reach site. • For us to come up with the Targeted Outcomes or Results in Level 4 we need to know what the volunteer and their Thai colleagues experience once the volunteer arrives, and we also need to see for ourselves the challenges the volunteers, counterparts, co-teachers, supervisors and field supervisors experience in the field. • This includes every aspect of the volunteer/That colleague experience: their work, their housing, their dress, the local government structure, the school structure, and so on. • As we continue to learn more and more about what it’s like at site, we can then create Targeted Outcomes/Results and then design our trainings from there.

29. Appreciate Inquiry in PhraeWhat I’ve Learned • Appreciate Inquiry Phrae: • Ms Chadarat (Education Supervisor Phrae) • Ms Jaloowan (co-teach w/Todd C. 113) • Ms. Antalee (co-teacher w/PC volunteer Barbara?) • Ms. Sunpatra Porn (co-teacher w/PC volunteer Ann and Fred Ceely) • Pee Goon (my co-teacher and co-teacher w/Derrick)

30. Appreciative Inquiry Phrae • What is good about having a Peace Corps volunteer? • Student can learn with a native English speaker • New teaching techniques & critical thinking skills • Learning about American culture, volunteers should bring pictures, teach about American holidays, sharing stories about American life, etc. • Lesson Planning • Pronunciation • Volunteers can translate local Thai stories into English which can then be used to teach English. • Teach grammar and English idioms to students and teacher.

31. Appreciate Inquiry Phrae • When I asked the teachers to rank what they noted in the previous slide they identified • FIRST: New teaching techniques & critical thinking skills • SECOND: Learning about American culture, volunteers should bring pictures, teach about American holidays, sharing stories about American life, etc. • Everything else they said was very important and would all rank about the same.

32. Appreciative Inquiry Phrae • Ideally with Appreciate Inquiry you would then be a vision statement. • Instead, the teachers and Ms. Chadarat wanted to talk to me about the problems they had with volunteers. • A point here. We preach and preach during PST1 that Thai people are not direct. THAT is not always the case and we need to find a way to explain in PST that Thai people are not all the same, neither are Americans. I laughed with Chadarat and the teachers because of their directness and agreed, let’s talk about the problems they had with volunteers.

33. Appreciate Inquiry Phrae • Problems/Challenges hosting a Peace Corps volunteer: • Volunteer not at school, or leaves early • Some only want to observe, not co-teach • Volunteers not at site, on vacation, going somewhere • School is too far away from the volunteer’s home • Volunteers do not have enough teaching experience/or techniques to share and do with the teacher. • Volunteers not follow Thai rules, or adapt to Thai culture • Lack of communication between the Thai teacher and volunteer.

34. Appreciate Inquiry Phrae • I want to note here that we had a discussion that many of their issues were some of the same issues volunteers in my PC Group 113 had with their co-teachers. • Ms. Chadarat expressed that she was going to request 3 new volunteers for Phrae. • To help with some of these issues we discussed that once a month the three volunteers and co-teachers could meet at the education office, plan the following months lessons, share what went well, what didn’t go well– this would further develop friendships, trust, and a spirit of sharing & learning between the co-teachers, volunteers and supervisor. This may be a model PC Thailand considers or refines.

35. Appreciate Inquiry Level 4 • Level 4 Targeted Outcomes must be based on experiences from the field. • Thus, a suggested Targeted Outcome I would have for PST1 based on the Phrae feedback would be • The TCCO volunteers will be able to use their co-teachers text book, create a 3 week lesson plan based on one of the chapters in the text book, set goals/objectives to be learned during the three week period, plan activities to reach those goals, and co-teach those activities with their co-teachers. • The TCCO volunteers will be aware that most Thai teachers have limited English language skills, inexperience pre-testing and testing for comprehension, and depend on games and songs to past the time in the classroom. • The volunteers will have ample time in PST1 to learn these concepts and practice these activities either in a Thai classroom or experiential learning environment so they feel confident when they reach site.

36. Informal Discussions with Volunteers at PST 2 • These conversations were held either between training sessions or while they were sharing beer after sessions in the evening– I had iced coffee! • I’m going to list a few of the issues the volunteers noted I feel PC Thailand can address through its PST1, PST2, Mid-Service, COS and other trainings.

37. Informal Discussions with Volunteers at PST 2 • Covering a lot of material during PST1 but not having enough time enough or opportunity to learn or apply what we have been taught. • Thai staff carries out many activities but at times does not have the understanding in how we can apply these activities at site. • Packet for site visit wasn’t complete, half didn’t do it, wasn’t not introduced or explained to us why it was important. • Not enough explanation as to why we are doing the activity and how to apply it at site. • Did not respect the overall intelligence of the group, a remedial slow pace, too much time spent on games and skits.

38. Informal Discussion with Volunteers at PST2 • Activities such as community mapping gave little instruction on how to analyze it or what to do with it later. • PCT Facilitated Discussions– PCT’s not skilled enough…no experience in Thai classroom. • The current PC volunteers who came to help with training did whatever they wanted to do . . . Some taught us how to get out of teaching English • Very little to no co-teaching opportunities during PST 1 or PST 2 • Too much time spent learning songs and games and not how to apply them in a systematic way which the Thai teachers need.

39. PST Informal Discussions with Volunteers at PST2 • Rules and regulations should be presented at beginning of PST. • Consistent follow through with any incident • Documentation of the incident that the volunteer sees and agrees that he/she broke • There should not be the fear that one will be kicked out for breaking a rule. Threatening volunteers with dismissal because they are honest on their evaluations is not fair. • Suggested Targeted Outcome: The volunteers will be made aware of all the rules, the negotiable and non-negotiable rules, and a clear discipline system.

40. Informal Discussion Sequencing • TCCO volunteers introduced to CAT second to last day of training. • PDM should be introduced to CBOD volunteers at PST2 • TCCO volunteers should concentrate on co-teaching, lesson planning, effective teaching activities etc during PST1 • TCCO volunteers should focus on further developing co-teaching techniques with teachers at PST2, with an introduction to PDM at PST2 or Mid-Service Training

41. Informal Discussion PST2 Cultural • No or little discussion of the difference between “rip roy” during training and at site– completely unprepared for this– I was scared to death I was going to get kicked out for dressing wrong. • Couldn’t be honest on evaluations for fear of being kicked out. • Cross cultural training– played too many games, spoke to us and treated us like children-- insulting • Girls should be allowed to wear pants during training; teachers wear pants…it’s the job I do that is important…that’s not message we got at PST1

42. Informal Discussion PST2 Cultural • Guys should wear khakis or slacks • Sexist: Girls can wear polo shirts out, but guys must tuck them in • Cleavage should be monitored • There should be a standard dress code addressed for 30 minutes and posted for all to see. • A consistent system should be put in place for infractions against dress code.

43. Informal Discussion PST2 Cultural • Suggestions: Peace Corps Thailand staff reconsider the anxiety that is caused by its cross-cultural, dress code approach. Do research at volunteers’ sites to see what is consider proper and improper. If a more conservative approach is still considered to be important to the PC Staff…it is ESSENTIAL the PC Staff explain to the volunteers that this is a conservative approach, and should adapt appropriately to what is worn at site. • Suggested Targeted Outcome: Dresscode is discussed day one, including above suggested explanation. Dress code is displayed at HUB, and a consistent discipline system is put in place. If a volunteer continues to violate dresscode, then perhaps Peggy or John need to be brought in, in order to evaluate the volunteer’s ability to adapt and if he/she should remain with Peace Corps Thailand.

44. PST 1 Combined Feedback • Early termination: needs to be clearly defined and explained at beginning of PST1. • There seeds to be a warning policy (with the exception of non-negotiable policies) • We were not sure if small infractions would end our service. • This stress was compounded with the stress of homesickness, culture shock and the ability to learn what we needed to learn during out training/language sessions. • “This was one of the sing-most detrimental aspects of our PST experience, and it can be addressed by having a clear policy for the early termination or trainees, and by making sure the policy is well-explained and understand at the early stages of PST.”

45. PST 1 Combined Feedback • “Facilitated sessions in which PC trainees discussed how they would react to different aspects of Thai culture made up a vast majority of cross-cultural training during PST. While a few of these sessions are helpful…we had little exposure to Thai culture, and our combined inputs were not often (helpful). As an alternative, these sessions need to learn more towards lecture-based classes, offering a description, a reasoning, and a history behind the Thai actions and traditions that trainees might find hard to understand. The more that the culture is understood, the easier it will be for volunteers to function within it.”

46. PACA tools were presented with worksheets. It would have been helpful to see how a successful PACA tool is used in the field and then guided in how to use one instead of it being left up to the volunteer to do the first time. • Debriefing of PACA tools included a lot of interesting information about the village; however, more information on what aspects or variations on the tools would have been more helpful. • Suggested Targeted Outcomes: Volunteers will be taught the PACA tools with ample time to carry out the activities in PST1 at first with assistance from PC Staff, and the on their own.

47. BACK TO LEVEL 4 • “Any training provided by staff or currently serving Peace Corps volunteer alike that cannot provide insightful and relevant information pertaining to these projects is using up our time. Great care needs to be taken to structure our training so that the information we need is provided to us as efficiently as possible.

48. Conclusion • The main idea here, for us to discover what our Targeted Outcomes should be, must come from the field. This is why Monitoring and Evaluation are so important. You can not improve or provide adequate training without knowing what is happening in the field.

49. Conclusion • The easy part is that we 3 opportunities to build on the volunteers and their counterparts knowledge: PST1, PST2, and Mid-Service Training. We can monitor first hand from the volunteers and their colleagues how things are going.

50. Conclusion • My last suggestion is this. • We have two programs: • CBOD with Jaree and Jaew • TCCO with Suvimon and Chadchay • Each program has about 40 participants. I think once a month, once every two months, every volunteer should be called in order to check in.