Our Lives in Evaluation: AEA Members’ Descriptions of Their Evaluation Work

1. Our Lives in Evaluation: AEA Members’ Descriptions of Their Evaluation Work Colleen Manning Leslie Goodyear Margaret Tiedemann

2. What would you say? “Imagine you are out to dinner with other AEA members, and each member is taking a minute or two to describe his or her evaluation-related work and/or study.  It’s your turn; what would you say?”

3. In the kitchen … AEA Value 5 AEA Value 4 AEA Value 3 AEA Value 2 AEA Value 1 Internal evaluation Responsive evaluation International development Capacity building Service orientation Many hats Involving stakeholders Technical assistance Social betterment Use Teaching New to evaluation Method-specific

4. Amuse-Bouche • “I am primarily a methodologist with over 40 years of experience in the major areas of evaluation from needs assessment to experimental clinical trials.  My major asset is that I have made most of the mistakes one can make at least once.” • “I have been doing program evaluation since 1992 without knowing it is what I am doing.” • “My favorite metaphor is the doctor of methodology (Dr. Meth) one, meaning that I tailor the best methodology available to unravel causes and effects of a program given the circumstances a program is embedded in.”

5. Antipasto Value 1:  We value ethically responsible, high quality, culturally responsive evaluation practices that lead to effective and humane organizations and ultimately to the enhancement of the public good. • “I work with people to help identify what works and what doesn't work, so that we can document what works and fix what doesn't.  If I'm lucky, I can do this very collaboratively, if not, I have to be sneaky and/or diplomatic.”

6. Antipasto • “I help organizations understand how evaluation can improve their processes and their work and achieve better results.  Evaluation can get everyone on the same page, working toward the same end.” • “I am a responsive evaluator focusing on facilitating the organizational development of groups who experience considerable social marginalization in American society.  I seek to help such groups develop their capacity for sustainability and growth.” • “In social terms, I'm interested in the contribution of evaluation to society, including validity, ethics, equity, and evaluation's part of the social contract.”

7. Antipasto • “I combine development of theory and evaluation practice to examine issues of social justice in the evaluation contexts in which I work. This provides me with a lens of cultural complexity needed to address issues that might be overlooked or misinterpreted if the evaluator is not sensitive to cultural diversity.” • “My work is typically in communities of color and I attempt to employ culturally responsive evaluation strategies in my work.”

8. Primo Value 2:  We value ethically responsible, high quality, culturally responsive evaluation practices that contribute to decision-making processes, program improvement, and policy formulation. • “Although my training is in statistics and measurement, I believe that every evaluation benefits from gathering qualitative evidence as well. Such information is almost always necessary to contextualize findings, and answer important questions about program implementation that will serve to improve the program.” • “I use the Baldrige Criteria for Performance Excellence to assess organizations’ performance, identify strengths and opportunities for improvement in their management systems, and prioritize issues for action to improve performance.” • “My work is classified as participatory and empowerment evaluation. I help my clients ‘walk the talk’ of their mission and vision statements by making data-informed decisions for continuous process improvement.”

9. Primo • “I am particularly interested in using evaluation as a decision support tool, making a contribution to evidence-informed public policy.” • “… Most of my clients are smallish nonprofits … Over time I build their capacity to institutionalize good data collection, analysis and reflection so that they improve their programs with or without a consultant's help.” • “Typically, my clients are trying to make important decisions or solve problems without enough information to go on. I help close those information gaps …”

10. Secondo Value 3:  We value a global evaluation community and understanding of evaluation practices. • "I'm working within a large, global NGO in their smallest sector -- domestic programming.  I work with a national team as well as eleven individual program sites across the U.S.  I do not have a degree and I rely heavily on the training from AEA and reading and research on my own..." • “It is rewarding in that the work we do usually has a genuine impact on the thinking and the strategies of a large, worldwide organization that plays a unique role in the global food security, agriculture, rural development, social change and poverty alleviation.” • “I work in global development evaluation.  From my home in the US or on short-term work visits overseas, I assist international organizations, public and private sector organizations and civil society organizations to assess implementation progress, program impact, or evaluability of programs.” • “I am doing evaluation in Serbia i.e. in the post-communist country which is currently going through an transitional (political, economical, social etc.) stage and where evaluation is not widely practiced nor recognized yet – nevertheless how extremely it is important and needed. These conditions are both quite challenging and very stimulating.”

11. Secondo • “My evaluation work is mostly with international NGOs and foundations. Besides providing high-quality work for the agencies I work for, I am committed to help improve the quality of evaluations conducted in the international development aid area…” • “I would share the opportunity that AEA offers for us to meet and share our work. I would then talk in detail about the methodological challenges in conducting evaluation in a very complex environment of international development assistance. I would ask for any best practices in the use of sound methods in conducting evaluations. I would share the mixed method approach that I take and the lessons learned in the process.”

12. Contorno Value 4: We value the continual development of evaluation professionals and the development of evaluators from under-represented groups. • “I'm learning evaluation as I go, and work alone without team members to bounce ideas off or learn from.” • “I am still in the learning phase but I'm fascinated with what each of you is doing.”

13. Contorno • “I would like to learn more about the nuts and bolts of biostatistical reasoning and qualitative methods.” • “Program evaluation is a new field for me. I have served as a business manager for approximately 20 years and have a strong interest in learning program evaluation. I look forward to attending the workshop Evaluation 101 in Baltimore.” • “I am a grad student and an eager to learn all I can. I want to make a living doing Evaluations and I want to be one of the best in my country (Barbados).”

14. Contorno • “I am continually learning the theories behind and the practical application of evaluation. What surprises me most is the number of people and various ways people evaluate.”

15. Dolce Value 5:  We value inclusiveness, multiculturalism, and diversity, welcoming members at any point in their career, from any context, and representing a range of thought and approaches. • “I lead evaluations of international programs focused on HIV/AIDS, healthcare, anti-trafficking, media capacity, education etc. A particular strength of mine is the application of appreciative inquiry to evaluation, but that is only one of the methods I use routinely.” • "Most of my evaluation experience has been related to health and social services programs, particularly for the Aboriginal peoples of Canada...currently working with Aboriginal representatives from small communities in the far north on developing "contextually appropriate" tools for evaluation of projects.”

16. Dolce • “I am new to the field of evaluation.  I direct a large office responsible for the treatment and educational programs for 24,000 inmates.  I am here to learn what works best to evaluate our programs for success and effective use of resources.” • “I am an entomologist working in the area of pest management.  I work with faculty across the state to develop evaluation methodology for research and outreach projects related to pest management...” • “My work is centered around empowering people to learn to use data to make decisions that advance their work.  I make sure that people feel a part of the evaluation process, and help them clarify their goals and values even as we work toward often rigid program goals.  I focus on helping people address barriers and challenges that hinder progress towards goals they find important, and on helping people feel good about the small changes that occur through their efforts.”

17. Some reflections …

18. “I am a Christian evaluator working for a Christian NGO.  I integrate spiritual disciplines in my evaluation of community development programs.” • ‘I try to uncover new knowledge to solve our country's most pressing social and economic problems.” • ‘I carry out public relations and publishing for the government evaluation agency that I work for.  I need to know about evaluation because I speak to groups around the country about the work of the organisation.  I don't carry out evaluations myself.” • ‘I purposely do not specialize in a specific sector as I enjoy the diversity and breadth of projects spanning public and non profit sectors.”

19. For more information • For more information about the member scan, go to: http://www.eval.org/Scan/aea08.scan.htm Thank you!