1 / 14

A Four-Tier, Question-Driven Evaluation Approach Rosalie Ruegg, TIA Consulting, Inc.

A Four-Tier, Question-Driven Evaluation Approach Rosalie Ruegg, TIA Consulting, Inc. Work prepared for the U.S. Department of Energy (DOE), under contract with Lawrence Berkeley National Laboratory (LBNL). Opinions expressed do not necessarily reflect those of the U.S. DOE or LBNL.

sherman
Download Presentation

A Four-Tier, Question-Driven Evaluation Approach Rosalie Ruegg, TIA Consulting, Inc.

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Four-Tier, Question-Driven Evaluation ApproachRosalie Ruegg, TIA Consulting, Inc. Work prepared for the U.S. Department of Energy (DOE), under contract with Lawrence Berkeley National Laboratory (LBNL). Opinions expressed do not necessarily reflect those of the U.S. DOE or LBNL. Opening the Black Box of Innovation: Interim Outcomes Gretchen Jordan, 360 Innovation LLC Jonathan Mote, George Washington University Rosalie Ruegg, TIA Consulting, Inc. American Evaluation Conference Washington DC, October 17, 2013

  2. Four-Tier, Eight-Step, Question-Driven Evaluation Approach: Overview • 4 tiers • 8 steps • Illustrative questions by tier • Methods & Metrics • Assessment of change, early outcomes, and need for mid-track changes Ruegg

  3. 4 Tiers Analyze Strengths & Weaknesses 3 2 4 1 Ruegg

  4. 4 Tiers/8 StepsTier 1: Describe Inputs, Activities, Targeted Industry, Outputs Ruegg

  5. 4 Tiers/8 Steps Tier 2: Assess Early Progress Ruegg

  6. 4 Tiers/8 StepsTier 3: Attribute Broader Interim Outcomes * A chapter of the report provides guidance on methods of assessing attribution, focusing on quasi-experimental study design or a counterfactual approach. Ruegg

  7. 4 Tiers/8 Steps Tier 4: Conduct Formative Analysis Ruegg

  8. Study Design & Mixed Analytical Approaches are Used to Answer Descriptive, Normative, and Impact Questions • Logic modeling • Industry/market analyses • Analysis of existing databases, records, and reports • Interview of experts • Net-chain analysis • Supply-chain analysis • Pre/post intervention comparisons • Comparisons of participants/nonparticipants • Econometric and statistical analyses • Counterfactual analysis if quasi-experimental design infeasible • Gap analysis Ruegg

  9. Tier 1 Questions RE Inputs, Activities, Targeted Industry, and Outputs How much, how many, who, what? • How much spent? • What mechanisms were used; activities carried out? • Who is participating • Patents, papers, prototypes, trained researchers, test facilities, research tools, ? • Alignment of market strategies with technical strengths? • Increased capital available? • Government purchases of early project? • Compared with baseline? • Compared with targets? Ruegg

  10. Tier 2 Questions RE Early Progress What changes are seen in 7 critical conditions affecting economic success? (1) new technical/market capabilities? (2) increased ability to attract funding? (3a) improved business practices? (3b) more favorable government policies? (4a) stronger networks? (4b) increased knowledge exchange? (5) value added? (6) entry of firms in value chain? (7) more robust domestic supply chains? Note: See preceding listing and discussion of 7 conditions by Gretchen Jordan. Ruegg

  11. Tier 3 Questions RE Assessment & Attribution of Broader Interim Outcomes What evidence is there that the DOE/EERE investment has caused – Ruegg

  12. Tier 4: Conduct Formative AnalysisWhat strengths, weaknesses, gaps, and barriers were found? Ruegg

  13. Summary • Conducting Tiers 1-3 (steps 1-6) will characterize the situation thus far: - goals, objectives, targets - resources expended - actions taken - participating organizations - outputs - state of relevant value chain - early progress - broader attributed outcomes • Conducing Tier 4 (step 7) will: - reveal if progress is in line with expectations - identify strengths, weaknesses, barriers, challenges - provide prel. diagnoses of what went right and what went wrong - set stage for discussion by DOE/EERE of implications & what-next response. Ruegg

  14. What is unique about this effort? Most retrospective studies focus on either: • Long-term economic impacts and return on investment or • Short- to long-term goal-related outcomes in physical units, e.g., energy saved, lives saved, tons of carbon averted, people trained, etc. In contrast, this Guide focuses on interim changes in a set of underlying conditions seen as critical to achieving long-term economic success Ruegg

More Related