Participant-oriented evaluation Approaches

1. Participant-oriented evaluation Approaches EDF5461 Summer 2011 Alison Moore Jenna Oster Aaron Paquette

2. Definition • Participant-Oriented Evaluation Approaches "currently include many different models, but their commonality is that they all use stakeholders- people with an interest or "stake" in the program- to assist in conducting the evaluation." (Fitzpatrick 189)

3. Key concepts • Involves participants in the program, managers, staff, and other key stakeholders (anyone involved with planning and implementation of the evaluation) as a key principle to good evaluation • Involve stakeholders to teach evaluators about program • Founded in Constructivism (humans gain knowledge and meaning from interaction and experience)

4. Data Collection methods • In Situ Observation • Observing stakeholders in their interaction with the program • Interviews • Talking to staff, managers and stakeholders to learn more about the program • Documents • Using program documents to help with the evaluation

5. Timeline 1967 A push to bring human elements and/or qualities into the evaluation process: make people/ stakeholders a part of the evaluation process Pre-1967 No desire to include the human elements and/or qualities within the evaluation process Mid-1970s Observations and identifying concerns and issues of the stakeholders were examined and became enwound with the P-PE evaluation 1975: Responsive evaluation Present Jean King: “…overarching term that involves program staff actively in decision making” Cousins and Earl: “Applied social research… partnership between evaluator and stakeholders” New methods of participatory evaluation continue to emerge

6. Major developers • Robert Stake: One of the first advocates for participant-orientated evaluation • The Countenance of Educational Evaluation (1967) • Allows for a wide range of data to be collected • A congruence between the intention and what is observed • This model looks for continuity between what happens and what is intended • Countenance Framework (1967) • Countenance Matrix • Responsive Evaluation • Provides qualitative evidence of a programs effectiveness • Includes stakeholders in the implementation of policies and goals

7. Major developers(continued) • EgonGruba and Yvonna Lincoln • Naturalistic Inquiry (1985) • Collects qualitative data to interpret and understand unique situations where there is a level of uncertainty • Reliability and Validity: “Trustworthiness” • Naturalistic Evaluation • Responsive Evaluation + Naturalistic Methodologies = Naturalistic Evaluation • Evaluators participate in the evaluation and try to uncover the needs of the all members • Three phases • Familiarizing Phase • Action Phase • Synthesis Phase • Fourth-Generation Evaluation (1989) • Usesconstructionist paradigm to gather, analyze, and organize evaluative material based on the needs of stakeholders • Never-ending process Credibility Transferable Conformability Dependable

8. Approaches Participatory evaluation: “An overarching term for any evaluation approach that involves program staff or participants actively in decision making and other activities related to planning and implementation of evaluation studies.” (p. 199) Two main types of participatory evaluation: P-PE T-PE Practical Participant Evaluation is used for practical reasons, limiting the evaluation to a particular organization. Transformative Participant Evaluation means just that, to transform and empower the stakeholders (social change).

9. Categories of Participatory Approaches • There are 3 dimensions in which participatory approaches differ, according to Cousins, Donohue, and Bloom (Fitzpatrick 200): • 1) Control over the evaluation or technical decision-making process • 2) Stakeholder selection • 3) Depth of Participation

10. Approaches (continued) Matrix (p. 204)

11. Strengths of participant-oriented evaluation approaches • When stakeholders are involved in evaluations, they . . . • . . . provide data sources, distribute surveys, and set up focus groups • . . . are familiar with program context and environment • . . . establish validity of the study because they possess/incorporate knowledge to inform decisions • . . . can communicate program situation/details to facilitate evaluator’s understanding • Including stakeholders in evaluations . . . • . . . gains their trust in the evaluator and the evaluation process • . . . leads to better chances of stakeholder use/adoption • . . . improves stakeholders’ understanding of program/organization

12. Limitations of participant-oriented evaluation approaches • Competence/feasibility • Is the evaluator truly capable of acting as facilitator, coordinator, and consultant during the evaluation process? • Are stakeholders able to perform tasks required of them? • Credibility • Because stakeholders are deeply invested in the program, will they act to change the program based on the evaluation results? • Political • Due to involvement of people (primary stakeholders, managers, staff) • Time-consuming and expensive

13. Review • Evaluators include stakeholders in evaluation planning and implementation • Stake and Gruba and Lincoln responded in the 1970s and 1980s to established methods of evaluation • Wanted approaches offering consideration of program stakeholders • Two main branches of participant-oriented approaches • Practical participatory evaluation (P-PE): encourage involvement/ownership, thus increasing use of results by stakeholders • Transformative participatory evaluation (T-PE): empower stakeholders by direct involvement in evaluation process • Three primary dimensions of participatory approaches • Evaluator or stakeholders possess control • Individual stakeholder selection • Extent of stakeholder involvement

14. Review(Continued) • Practical participatory (P-PE) approaches often rely on • Stakeholder involvement • Practical participatory evaluation • Developmental evaluation • Transformative approaches often rely on • Empowerment evaluation • Deliberative democratic evaluation • Many definitions for participative evaluation exist • All involve stakeholders, but no additional details are standard