Interactive Evaluation Practice

1. Interactive Evaluation Practice Presenters: Jean A. King Laurie Stevahn University of Minnesota Seattle University Organizational Leadership, Educational Leadership Policy, and Development Doctoral Program kingx004@umn.edustevahnl@seattleu.edu (612) 626-1614 (206) 296-2559

2. Overall Session Goals • Examine frameworks that ground Interactive Evaluation Practice (IEP). • Experience strategies for facilitating IEP and rationales for using them in evaluation studies. • Consider implications and applications of the strategies for your own evaluation practice.

3. Shape of This Afternoon • Introductions —you, us, we • Frameworks —for Interactive Evaluation Practice • Evaluator’s Dozen of Cooperative Strategies —responding to set content —generating information —organizing information

4. Jean’s background . . . • Roles►teacher, teacher educator, director of a collaborative research center, professor of evaluation studies • Education► English, curriculum and instruction, research design • Specialties►participatory evaluation, evaluation capacity building, evaluator competencies • Passions►family, friends, program evaluation, kittens/cats (anything that “meows”)

5. Laurie’s background . . . • Roles► teacher, consultant, researcher, evaluator, professor • Education► political science (BA), curriculum and instruction (MEd), educational psychology (PhD) • Specialties► cooperative strategies, constructive conflict, evaluator competencies, organizational change, inquiry methods • Passions► faith, family, friends, music, art, creativity, culture, collaboration, travel

6. Your background . . .

7. Strategy #1: Voicing VariablesHow long have you been involved with evaluation? • Less than a year • 1-5 years • 6-10 years • More than a decade

8. Strategy #1:Voicing VariablesIn what fields/contexts do you work? • Healthcare • Education • Social service • Government • Nonprofit • International • Other . . .

9. Strategy #1: Voicing VariablesYour role as an evaluator . . . • Internal • External • Both

10. Strategy #12: Fist to FiveWhat is your experience with . . . • Qualitative studies • Quantitative studies • Mixed-methods studies

11. Strategy #12: Fist to FiveWhat is your experience in conducting . . . • Single-program/organization evaluations • Large-scale and/or multiple-site evaluations • Community development and/or grass-roots evaluations

12. Strategy #5: Round-Robin Check-In • Form groups of three • Introduce yourself • Your name • Why you want to learn about interactive strategies

13. Grounding Frameworks for IEP • Basic Inquiry Tasks (BIT) • Interactive Participation Quotient (IPQ) • Evaluation Capacity Building (ECB)

14. FIRST FRAMEWORK — Exhibit 2.1 (p. 23)Basic Inquiry Tasks (BIT) • Framing questions (focusing the study) • Determining an appropriate design • Identifying samples (sources of information) • Collecting data • Analyzing data and presenting results • Interpreting results • “Reporting”

15. What are possible evaluator-client roles and relationships in carrying out BIT? • A relationship exists between the evaluator and client—i.e., program leaders, staff, funders, community members, other evaluation stakeholders • Involvement in evaluation decision making and implementation may shift between the evaluator and client/stakeholders during the study

16. HIGH Evaluator Involvement in decision making and implementation Program leaders, staff, community members LOW Evaluator-directed Collaborative Participant-directed ZONES SECOND FRAMEWORK — Exhibit 2.3 (p. 27)Interpersonal Participation Quotient (IPQ)

17. Diverse evaluator roles . . . • Technical expert on evaluation research design, measurement, coding, data analysis • Facilitator of group interaction • Coach of others doing their own evaluations • What else?

18. THIRD FRAMEWORK — Exhibit 2.6 (p. 35)Evaluation Capacity Building (ECB)

19. IEP highlights the importance of . . . • Involving people effectively— constructive interaction with others is essential to the success of all evaluations • Learning through evaluation— interaction and participation should promote learning • Building capacity to think evaluatively— through meaningful engagement in evaluation planning, acting, reflecting

20. An evaluator’s dozen of interactive strategies . . . #1. Voicing Variables #2. Voicing Viewpoints/Beliefs #3. Choosing Corners #4. Cooperative Interviews #5. Round-Robin Check-In #6. Making Metaphors

21. An evaluator’s dozen of interactive strategies . . . #7. Data Dialogue #8. Jigsaw #9. Graffiti/Carousel #10. Concept Formation / Cluster Maps #11. Cooperative Rank Order #12. Fist to Five #13. Dot Votes / Bar Graphs

22. Different Types of Involvement An evaluator’s dozen of interactive strategies for . . . • Level I Responding to set content (#1-3, #8, #10-13) • Level II Generating information (#4-7, 9) • Level III Organizing or sharing information (#8, #10-11, 13)

23. Strategy #2: Voicing Viewpoints/Beliefs A. A good evaluator shares control of the study with clients from start to finish. 1 Strongly 2 Agree 3 Disagree 4 Strongly Agree Disagree ____________________________________________________________________________________________________________________________ B. Program participants will bias a study if they are involved in planning it. • 1 Strongly 2 Agree 3 Disagree 4 Strongly Agree Disagree ____________________________________________________________________________________________________________________________________________________________ C. Program staff must be neutral during an evaluation study of their own programs. • 1 Strongly 2 Agree 3 Disagree 4 Strongly • Agree Disagree

24. Strategy #13: Dot Votes / Bar Graphs A. A good evaluator shares control of the study with clients from start to finish.

25. Strategy #13: Dot Votes / Bar Graphs B. Program participants will bias a study if they are involved in planning it.

26. Strategy #3: Choosing Corners C. Program staff must be neutral during an evaluation study of their own programs.

27. Strategy #4: Cooperative (three-step) interview • Three roles create three steps: • Interviewer • Responder • Recorder • The interview process is structured to build on psychological principles of cooperation

28. Interview FocusTEAMWORK—it was terrific! nInterviewer → seek input (respectfully) • What made the experience so successful? • When, where, who, what, why, how? nResponder → tell your story nRecorder → document (key words/phrases) _______________ ROTATE ROLES

29. Similarities/Themes ________________________________ What does this mean for evaluators?!

30. Strategy #8: Jigsaw • Home groups divide information/readings (Persons A, B, C, each get a different part) • Expert groups of two (all A’s, B’s, C’s) read and prepare to teach • Back to home groups to present • Apply the entire body of information

31. Jigsaw Scenario . . . • Large social service organization • Provides numerous diverse programs for the community • Assessing its current mission/vision/values in light of changing social, economic, political, educational, and technological concerns • Evaluators hired to conduct ongoing meetings with groups of stakeholders who are all in for the long haul • “Pluses/Wishes” meeting reflections from . . . • Program Directors (gold sheet) • Program Providers (blue sheet) • Program Recipients (green sheet)

32. Jigsaw process . . . nHOME TEAM of three—each member gets a different segment (gold, blue, or green sheet) nEXPERT PAIR of two—find one other person in the room who has your same segment (two golds together, etc.) nRead the data and identify major themes nReturn to HOME TEAM . . . • Share major themes from each stakeholder group • Compare/Contrast across all stakeholder groups • Recommendations for future action?!

33. Strategy #9: Graffiti / CarouselStrategy #10: Concept Formation • Can be done on the wall, informally, and is then called graffiti • Can be done on flipchart paper passed among groups and is then called carousel

34. Strategy #9: Graffiti / Carousel 1. Face-to-face communication is most useful when . . . 2. Face-to-face communication is frustrating when . . . 3. Electronic communication works well when . . . 4. Electronic communication is problematic when . . .

35. Strategy #10: Concept Formation _______________________________ 1. Write one comment per sticky note, as many as possible. _______________________________ 2. Organize “alike” items into clusters/groups/themes. _______________________________ 3. Label each cluster/group/theme.

36. Strategy #11: Cooperative Rank Order • An interactive social process for reaching consensus on a rank order • Sequence from most to least effective, best to worst option, highest to lowest priority, or some other continuum. • Colored paper (strips) can facilitate cross-group comparison

37. Strategy #11: Cooperative Rank Order Evaluators need support, too! • Consider a list of practices to support the professional development of evaluators • Sequence the practices from most to least helpful • Be ready to explain/defend your reasoning

38. Ongoing opportunities for formal professional development near-by (e.g., attending trainings, local conferences) Opportunities to attend national conferences (at least one a year) Regular informal gatherings with other evaluators to discuss issues, problem solve concerns, share “what works,” etc. Reading professional literature on your own Participating in an informal evaluation “book club”

39. Strategy #7: Data Dialogue • A process to use when you cannot afford focus groups • It takes advantage of some of the processes of the three-step interview • Can be useful in community settings

40. Strategy #7: Data Dialogue • Purpose To provide input on today’s workshop. • Directions Form groups of 3-4; decide who will write; make sure all input is recorded (this is NOT about consensus; it’s about getting everyone’s thoughts—seek and expect diverse perspectives). • Results Your input will be kept confidential and will NOT be traceable back to you. • Participants Please indicate that you participated by completing the information below, then detach this top sheet and place in the envelope for confidentiality. Do NOT put names on any other sheets. Thank you for participating! Signatures: __________ __________ __________ __________

41. Strategy 7: Data DialogueTOPIC: Today’s Session

42. Strategy #8: Making metaphors “A is worth 1000 words.” ___________________________________________________________________________________________________________________________________ Tall impressive terrifyingBIG magnificent

43. Find a picture to finish this phrase . . . “Interactive Evaluation Practice will be successful when ________.”

44. An evaluator’s dozen of interactive strategies . . . #1. Voicing Variables #2. Voicing Viewpoints/Beliefs #3. Choosing Corners #4. Cooperative Interviews #5. Round-Robin Check-In #6. Making Metaphors

45. An evaluator’s dozen of interactive strategies . . . #7. Data Dialogue #8. Jigsaw #9. Graffiti/Carousel #10. Concept Formation / Cluster Maps #11. Cooperative Rank Order #12. Fist to Five #13. Dot Votes / Bar Graphs

46. Review- Overall Session Goals • Examine frameworks that ground Interactive Evaluation Practice (IEP). • Experience strategies for facilitating IEP and rationales for using them in evaluation studies. • Consider implications and applications of the strategies for your own evaluation practice.

47. Thanks! Jean & Laurie

48. Thanks!