PLT Program Evaluation

1. What do funders want to know and can you (provide the data to) give it to them? Norie Dimeo-Ediger Carla Rapp Michelle Youngquist PLT Program Evaluation

2. We would like to know that your forest education programs for educators (PLT workshops, tours, etc.) are effective and a good investment of our money. $$$$$$$$$$$$ Here’s what our stakeholders said

3. Conduct an extensive (and expensive) statistically valid evaluation. Here is what we heard

4. Here is what we thought

5. I know that you believe you understand what you think I said, but I’m not sure you realize that what you heard is not what I meant. Robert McCloskey

6. What did the stakeholders mean? ????????? So………

7. Norie asked Pete and Paul. • Carla asked Larry, Winston and Bill. • Michelle asked Kevin, Bob and Mike.

8. What did they say????

9. I want to know that teachers and students “get it.” • -It would be nice to be able to track from grade to grade and year to year the attitudes and understanding of the students regarding our issues. • -I want you to go out and tape the teachers using the activities so we can see them in action.

10. Are they using the materials? • Has it changed (for the better) the way they teach about forests?    • I want to know enough people have responded to the questionnaire/survey that I can trust the results • Is PLT still the best way to deliver our message and how do we evaluate that?

11. How do we do what they want?

12. Level 1. Reaction – What is the participants’ response to the project or activity? • Level 2. Learning – What did the participants learn? • Level 3. Behavior or Application – Did the participants’ learning affect their behavior? • Level 4. Results – Did participants’ behavior change move the original situation towards the objective • Source: Kirkpatrick (1994). • Designing Evaluation for Education Projects – NOAA Levels of Evaluation

13. Level 1. Reaction – What is the participants’ response to the project or activity?

14. When asked to rate their content knowledge before/after the workshop the responses were: • BEFOREHigh54321Low 1 5 8 10 5 1 • AFTER High54321Low 17 13 0 0 0 Level 2. Learning – What did the participants learn?

15. Level 3. Behavior or Application – Did the participants’ learning affect their behavior?

16. Oil City is a Title I school in rural northwest Louisiana. In 2001, faced with declining enrollment and a possible closure, teachers and the administration worked together to create a school with an environmental focus. Measurable Achievement The success of the new environmental education focus and the activities that sprang from it brought recognition to the school. In 2004–2005, the school was recognized by the Louisiana Association of Business and Industry as one of the “Top 10” most improved schools in Louisiana. In 2006, the school received the National School Change Award sponsored by Fordham University Graduate School of Education, the American Association of School Administrators, and Pearson Education. Students have been invited to perform at the Louisiana Forestry Association Conference in Shreveport. In 2006, a fourth-grade teacher at the school, Brenda Smith, was named a National Outstanding Educator for Project Learning Tree. Level 4. Results – Did participants’ behavior change move the original situation towards the objective

17. Focus groups • Questionnaires or surveys • Observation • Case studies Some evaluation methods that might answer their questions

18. Purpose-to explore a topic in depth through group discussion • Advantages-key info can be conveyed, efficient way to get range and depth of info • Disadvantages-can be hard to analyze, need good facilitator, can be hard to schedule Focus Groups

19. Purpose-to quickly/easily obtain a lot of info from people in a non-threatening way. • Advantages-anonymous, inexpensive, large sample size, easy to compare and analyze • Disadvantages-might not get careful feedback, wording can bias response, may not tell full story, low return rate Questionnaires and Surveys

20. Purpose-to gather accurate info on how people behave • Advantages-generates data about behavior not reported behavior, data collected in context • Disadvantages-time intensive, “observer effect”, small sample size Observation

21. Purpose-to conduct comprehensive examination through cross comparison of cases • Advantages-provides vivid imagery, and detailed understanding of the experience • Disadvantages-time intensive, detailed understanding of a particular participant results can not be generalized. Case studies

22. What? To whom? How much? How often? What language? eg. Edu-speak? • Communicating results should not be an after thought, but part of the evaluation plan. Communication of results

23. Carla: Pre/Post “Viewpoints on the Line” adaptation, video-taping teachers doing PLT • Norie: Professional 3rd party evaluation • Michelle: Surveys, portfolio of teacher & student work. Collecting, but: • You do need to communicate it to the stakeholders. How best to do that? Some examples of what we’ve done

24. What have others experienced

25. Thank you! • Norie Dimeo-Ediger, Oregon Forest Resources Institute dimeo-ediger@ofri.org 971-673-2956 www.OregonForests.org • Carla L. Rapp, Georgia Forestry Association carla@gfagrow.org 478-992-8110 www.gfagrow.org • Michelle Youngquist, Idaho Forest Products Commission plt@idahoforests.org 208-334-4061 www.idahoforests.org Questions?

26. Carla, Michelle & NORIE