180 likes | 326 Views
Les Bolt, Ph.D. Appalachian State University 2009 National Evaluation Institute Louisville, KY October 10, 2009. Evaluation Realities: What the Theories and Standards Do Not Say. Researcher Context. Ph.D. in Research and Evaluation from UVA 30+ Years Experience Doing Formal Evaluation
E N D
Les Bolt, Ph.D. Appalachian State University 2009 National Evaluation Institute Louisville, KY October 10, 2009 Evaluation Realities:What the Theories and Standards Do Not Say
Researcher Context • Ph.D. in Research and Evaluation from UVA • 30+ Years Experience Doing Formal Evaluation • 75+ External Third-Party Evaluations • 1-4 Year Terms • Local/Regional/State/National Scope • $5,000 - $15,000,000 Budget Range • Public/Foundation/Non-Profit/Private Monies • Department/College/University Level Experience in Personnel and Program Evaluation, and Student Assessment
Research Methodology • Narrative Review • Evaluation Narratives • Evaluator’s Notes • Client Feedback • Auto-Ethnographic Reflection • Evaluator Interviews
Major Hurdles That Emerged • Insufficient Clarity of Outcomes • Lack of Baseline Data and Data Sources • Conflicting Stakeholder Expectations • Inertia from Previous Program/Practices • Fluid Leadership
Insufficient Clarity of Outcomes • Process Focus Instead of Outcome Focus • Fuzzy Outcomes • Processes and Outcomes Not Logically Related
Lack of Baseline Data and Data Sources • No Reliable Pre-Measures • Pre-Measures not Related to Outcomes • No Data Available to Construct Baseline
Conflicting Stakeholder Expectations • Not All Stakeholders Involved in Project/Program Conception • Stakeholder Drop-In / Drop-Out • Expectations not Expressed at Beginning of Project
Inertia from Previous Programs/Practices • Laundry List Effect • Processes and Materials Not Clearly Defined • “Resentful Demoralization” Surfaces
Fluid Leadership • Concept People Effect (Getting $$ vs. Operating Project) • Shifting Leadership • Loss of Leadership Interest
Managing the Hurdles • Recognize • Assess Impact • Accommodate • Acknowledge Limitations
Strategies for Accommodating Hurdles • Conduct Environmental Scan before Beginning • Triangulation of Expectations • Baselines Based on Best Practices Research • Establishing Firewalls between Previous and Current Practice • Identification of Specific Changes in Governance, Staffing and Funding
Conduct Environmental Scan before Beginning • Stated Processes vs. Actual Practice • Stated Processes vs. Resource Allocation • Stated Outcomes vs. Data Collected • Rationale for Discrepancies
Triangulation of Expectations • Identification of “Real” Stakeholders • Communication of Expectations • Consensus on Expectations • Consensus on Data Needed
Baselines Based on Best Practices • Understand the Project/Program Limitations • Review Literature that Applies • Plan and Budget for Best Practices Research • Do Your Homework
Establishing Firewalls between Previous and Current Practice • Set Clear Project Dates • Be Clear with Definitions • Use Qualifying Prompts in Surveys, Questionnaires, etc. • Identify Sources of Bias
Specific Changes in Governance, Staffing and Funding • Look for Explicit Policy Changes • Look for Budget Changes and Transitions to Hard Monies • Look for Personnel Shifts and Changes in Job Descriptions • Look for Hard Resource Allocations
Summary • Understand Context • Understand the Program or Project • Be Flexible but Rigorous • Ask a Lot of Questions • Report What You See
Questions and Discussion Contact: boltll@appstate.edu or boltll@comcast.net