Meeting an Evaluation Challenge: Identifying and Overcoming Data and Measurement Difficulties

1. Meeting an Evaluation Challenge: Identifying and Overcoming Data and Measurement Difficulties AEA Evaluation 2006 “The Consequences of Evaluation” RTD TIG, Think Tank Session Portland, Oregon November 4, 2006 Rosalie T. Ruegg Managing Director TIA Consulting, Inc. ruegg@ec.rr.com Connie K.N. Chang Research Director Technology Administration U.S. Department of Commerce cchang@technology.gov

2. The 4th in a Series of Think Tanks on Barriers to Evaluation • 2003: Identification of 6 Types of Barriers to Evaluation • 2004: Focus on Institutional and Cultural Barriers—Feedback Loops • 2005: Focus on Methodological Barriers • 2006: Focus on Data and Measurement Difficulties 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

3. Overview • Six Categories of Barriers Identified -- 2003 • Institutional/cultural -- 2004 • Methodological -- 2005 • Resources • Communications • Data/Measurement -- 2006 • Conflicting stakeholder agendas 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

4. 2003 Think Tank found … Striking commonality of evaluation barriers among programs and across countries 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

5. These barriers were said to impede … • Demand for evaluation • Planning and conducting evaluation • Understanding of evaluation studies • Acceptance and interpretation of findings • Use of results to inform • program management • budgetary decisions • public policy 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

6. 2006 Think Tank focus – data and measurement difficulties Data difficulties • Trail gone cold • Missing data • Data quality • Other? Measurement difficulties • Incommensurable effects • Forecasts for prospective analysis • Aggregation across studies • Inadequate treatment of uncertainty and risk • Accounting for additionality and defender technologies • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

7. Data difficulties • Trail gone cold • Missing data • Data quality • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

8. Difficulty: Trail gone cold With long time gaps between research, results, and evaluation, evaluators find … • Memory lapses • “Over the transom” effect with trail broken • Mixture of funding sources • Distinctiveness lost with technology integrations • Departure of key employees • Acquisition, merger, death of companies • Other? (Generated by Think Tank discussion) • Use of a financial instrument that does not have a legal requirement or dedicated budget to stimulate reporting/cooperation with evaluators • Reliance on a partner to report without regard to capability • Mechanics of surveying may be a problem 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

9. Dealing with trail gone cold • Your thoughts? (Generated by Think Tank discussion) • Build in requirement to report • Be proactiveness in conducting surveys a few years post project (e.g., Tekes – 3 yrs out; 67% response rate; no big changes between 3 yrs and 5 yrs out) • Incentives to high response rates are key because 5% of projects account for 95% of economic gains that random sampling would miss 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

10. Dealing with trail gone cold—Overview • Third best: Conduct “archeological digs” • Second best: Use clues to focus “detective work” • Best: Be proactive. Track and document data in real time 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

11. Difficulty: Missing Data • Data collection is spotty • Responses are incomplete • Files are corrupted • Not all paper records have been converted to electronic files • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

12. Dealing with missing data • Your thoughts? (Generated by Think Tank discussion) • Compose data from several sources to fill gaps. • If questions are too difficult to respond to – not reasonable, or confidential – you end up with missing data. So, do trial testing before survey launch. • If program is too young to have data, use proxy data, e.g. ATP used Japanese data to test a firm productivity model, and later ran it with ATP data and got similar results. • Explore statistical techniques to impute missing data. • Use security to ensure staff are not taking or corrupting data. • Look for pattern of missing data. • Look for biasing effects in data collection. • Check for errors in transcribing data from paper to electronic records (e.g., a check of one survey found tabulated 200 pregnant men!) 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

13. Dealing with missing data—Overview • Prevention is the best approach: implement sound data collection • Find the missing data using multiple strategies • Use proxy data • Use techniques for dealing with partial data • Use techniques to impute data Forthcoming book: Missing Data by Patrick McKnight, and Katherine McKnight (George Mason U); Souraya Sidani (U of Toronto); and Aurelio Jose Figueredo (U of Arizona) 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

14. Difficulty: data quality issues • Are the data valid for the intended use? • Other? (Generated by Think Tank discussion) • Source is key to data quality (i.e., are we asking the right people; are they motivated to answer truthfully?) • Are data used for making comparisons comparable? • Has a thorough definitional effort been conducted prior to data collection to ensure the right data are collected? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

15. Dealing with data qualityissues • Your thoughts? (Generated by Think Tank dicussion) • Think through upfront, scope out what you want to collect; how you go about collecting (e.g., NEDO thought through data collection; at the same time, flexibility is important – to avoid locking-in too early and to allow for adjusting/modifying/correcting – i.e., to support an iterative process in data design) • Avoid overly complex data collection instruments • Check data entry to detect human error • Provide means of calibrating answers and spotting outliers 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

16. Dealing with data qualityissues—Overview • Understand what is required by detailed up-front exploration • Assess reliability of data processes • Use data quality assurance tools • Monitor data quality over time • “Clean” data • Standardize data to conform to quality rules • Verify calculations [Possible Sources: International Association for Information and Data Quality (IAIDQ); Data Management Association (DAMA)] 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

17. Measurement difficulties • Incommensurable effects • Forecast for prospective analysis • Aggregation -- across studies, projects, levels, etc. • Inadequate treatment of uncertainty and risk • Accounting for additionality and defender technologies • Other? • Instrumentation, who does the measurement? How does calibration occur? • Problem of attribution is profound, scientists argue over it (discovery, use), and problem increases exponentially closer to commercialization (lack of acknowledgement of significance of competitor’s work). Reluctance to give government credit (e.g., companies don’t want gov’t to recoup; want to deny gov’t support helped create success today). • Double counting • Difficulties in measuring innovation 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

18. Difficulty: Incommensurable effects Presenting effects measured in different units in a single study • Knowledge – # papers, patents • Economic -- $ • Environmental – level of emissions • Safety -- # accidents • Employment -- # jobs • Energy security – barrels of oil imports • … 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

19. Dealing with incommensurables—some ideas • In some cases, you can express different effects in a common measure (e.g., make them commersurable) • In other cases, decision makers may want effects to be expressed separately in their own units • In which case, the decision maker must make trade-offs subjectively • Or, the evaluator weights and combines different effects using index values that are easier to compare, e.g., ATP’s Composite Performance Rating System (CPRS) • Or, other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

20. Difficulty: Forecast for prospective analysis • More uncertainties compared with ex-post analysis • technical uncertainties • resource uncertainties • market uncertainties (market size, timing, speed of commercialization) • other … • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

21. Dealing with forecast for prospective analysis—some ideas • Build uncertainty into the estimations • Consider all important alternatives • different levels of technical success • different market applications to consider • Revise forecast as additional info becomes available • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

22. Difficulty: Aggregation across studies • Different base years • Different time periods • Different methods • Differences in underlying assumptions, models, algorithms • Different types of effects measured • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

23. Dealing with aggregation across studies—some ideas • Best is to reduce incompatibility by standardization, and require transparency and replicability • Where there is internal consistency, combine common measures across studies • In place of aggregation, summarize across studies in terms of a single measure (e.g., a table of IRRs), at your own risk! • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

24. Difficulty: Inadequate treatment of uncertainty and risk, leading to -- • Overstatement of results • Unrealistic expectations • Faulty decisions • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

25. Dealing with inadequate treatment of uncertainty and risk Use a techniques such as the following: • Sensitivity analysis • Statistical test of variation (e.g., confidence interval) • Expected value analysis • Decision trees • Risk-adjusted discount rates • Certainty equivalent technique • Computer simulations using random draw across range of values • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

26. Difficulty: Accounting for additionality and defender technology • Incorrectly attributing all observed changes to a program’s effect • Partial identification or double counting of additionality effects • Problems in defining reference groups for control studies • Recognizing limitations of counterfactual questions (i.e., non-experimental design) • Ignoring or incorrectly modeling the defender technology • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

27. Dealing with additionality and defender technology—some ideas • Always consider effects with and without the program (e.g., counterfactuals, before/after comparisons, control groups) • Breakout additionality effects into component parts 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

28. Dealing with additionality and defender technologies, cont’d • Systematic comparison prob=10% Success rate A prob=50% Success rate B prob=40% Program--yes Success rate C Success rate D Defender tech with improvement rate X Defender tech with improvement rate Y Program--no “Dynamic modelling of defender tech” 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

29. 2006 Think Tank focus – data and measurement difficulties Data difficulties • Trail gone cold • Missing data • Data quality • Other? Measurement difficulties • Incommensurable effects • Forecast for prospective analysis • Aggregation across studies • Inadequate treatment of uncertainty and risk • Accounting for additionality and defender technologies • Other? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

30. Summary • Six Categories of Barriers Identified -- 2003 • Institutional/cultural -- 2004 • Methodological -- 2005 • Resources • Communications • Measurement/data -- 2006 • Conflicting stakeholder agendas • Nov 2007 … what’s next? 2006 AEA Evaluation Conference Think Tank Ruegg & Chang

31. Rosalie T. Ruegg Managing Director TIA Consulting, Inc. ruegg@ec.rr.com Connie K.N. Chang Research Director Technology Administration U.S. Department of Commerce cchang@technology.gov Contact information 2006 AEA Evaluation Conference Think Tank Ruegg & Chang