Follow-up Monitoring and Outcome Survey for National R&D Projects at NEDO

1. November 10, 2007 AEA Conference at the Sheraton Hotel Baltimore, Maryland Follow-up Monitoring and Outcome Survey for National R&D Projects at NEDO Presenters: Hiroyuki Usuda usudahry@nedo.go.jp Kazuaki Kohmotokohmotokza@nedo.go.jp Chairperson: Momoko Okadaokadammk@nedo.go.jp

2. Contents • Introduction of NEDO and Evaluation Practices (10 min.) • Part Ⅰ-Study of the Relation Between Ex-post Evaluation • and Follow-up Monitoring of National R&D (15min.) • Introduction of Follow-up Monitoring and Evaluations (5min.) • Part Ⅱ-Study of Important Management Factors Based on • Follow-up Monitoring Data (15min.) • Part Ⅲ-An Approach to Understanding Outcomes of National R&D • in Energy Conservation Project (15min.)

3. What’s NEDO?

4. New Energy and Industrial Technology Development Organization Japan’s largest public R&D management organization

5. History of NEDO • -Established in October 1980 as a semi-governmental organization under the supervision of METI. • - Original activities : • 　　 ・National R&D projects on new energy • -Current activities : • ・Support of industrial technology R&D (since 1988), • ・Support of environmental technology R&D (since 1990), • ・Introduction and promotion of new energy and energy conservation • technology (since1993), and • ・Acquisition of Kyoto Mechanism Credits (since 2006). • -NEDO became an Incorporated Administrative Agency in October 2003 as part of the process of overall government reforms.

6. Organization of NEDO Chairman, President & Executive Directors Auditor Headquarters General Management Depts. General Affairs; Personnel Affairs; Accounting; Assets Management; Inspection and Operational Management; and Information and Systems. Electronic & Information Technology Development Dept. Nanotechnology & Materials Technology Development Dept. Biotechnology & Medical Technology Development Dept. Policy Planning and Coordination Dept Machinery System Technology Development Dept. R&D Project Evaluation Dept. Fuel Cells & Hydrogen Technology Development Dept. R&D Promotion Dept. Energy and Environment Technology Center Energy and Environment Policy Dept. New Energy Technology Development Dept. Energy Conservation Technology Development Dept. Staff : approximately 1,100 Budget : US$1.9 billion (FY2007) Environment Technology Development Dept. Kyoto Mechanisms Promotion Dept. Coal Projects Dept., Coal Mine Subsidence Dept.

7. Others Kyoto Mechanism Credit Activities 4% 6% Research and Development Activities Introduction and Dissemination Activities 69% $1.3 billion 36% Budget Planning in FY 2007* R&D Support of NEDO • -R&D support is NEDO’s core activity. • Number of NEDO’s R&D projects in FY2007 is about 110. • NEDO funds high-risk R&D • NEDO creates broad networks *some items are overlapping: all items added up do not match the 100%

8. Environment Nanotechnology and materials New energy NEDO’s R&D Technology Areas Energy conservation Overview of R&D Projects Length: 3~5 years, at longest 10 years or more Budget: approximately$ 8 million / 5 years Number of projects: about 130 / year Project participants: mainly big companies, partly, medium and small companies, universities, public agencies Form of contract: commitment / subsidies Fuel cells and hydrogen Electronics and information Machinery systems Biotechnology and medical technology Industrial Technology Field Energy Technology Field Contributions to the practical application of technology

9. How Does NEDO Evaluate R&D Projects ?

10. Purpose of R&D Project Evaluations NEDO’s Successful Achievements Accessible Projects Easy-to-understand Information Plan Planning and Assessment Optimal Project Management See Technology Evaluation Do Project Management R&D Project Evaluation Dept. R&D Project Evaluations are an important part of R&D project management and serve several purposes: -To reflect economic and social needs on R&D activities; -To implement R&D projects effectively and efficiently; -To improve the overall project management.

11. 5 Principles of Evaluation at NEDO Transparent -Publicly disclose evaluation results, methods, and reflections. Explicit -Encourage discussions. Effective -Adopt evaluation methods for management improvement. Neutral -Utilize external evaluators. Efficient -Simple evaluation procedures.

12. Short-term Outcomes Mid- and Long-term Outcomes Outcome Survey R&D Project Evaluations at NEDO Ex-ante evaluation At R&D theme selection Midterm evaluation In the middle of project term Ex-post evaluation 1 year following project completion Follow-up monitoring & evaluation For 5 years after project completion ▼ ▼ ▼ ▼ Project implementation Post-project activity

13. Study of the Relation Between Ex-post Evaluation and Follow-up Monitoring of National Research and Development(Part 1) November 10, 2007AEA Conference Hiroyuki USUDA New Energy and Industrial Technology Development Organization (NEDO)

14. OUTLINE - R&D EVALUATION SYSTEM AT NEDO(Overview) -EX-POST EVALUATIONS -FOLLOW-UP MONITORING & EVALUATION - Objectives of the study - Data - Results - Conclusion

15. R&D EVALUATION SYSTEM AT NEDO (Overview) - EX-POST EVALUATIONS - FOLLOW-UP MONITORING & EVALUATION

16. R&D Project Evaluations Ex-ante evaluation At R&D theme selections Midterm evaluation In the middle of project term Ex-post evaluation 1 year following project completion Follow-up monitoring & evaluation For 5 years after project completion ▼ ▼ ▼ ▼ Project implementation Post-project activity

17. Ex-post Evaluations - Each project is assessed and evaluated in term of the following 4 categories : ・ Purpose and strategy ・ Project management ・ R&D achievements ・ Prospects for commercialization

18. Intermediate & Ex-post Evaluations average Categories for evaluation ratings (score) 1. Purpose and strategy 2.3 A A B B B B 2. Project management 1.8 A B C B C B 3. R&D achievements 1.5 B C C C B B 4. 1.2 Prospects for commercialization B B D C C C yes Go to 2. No Not enough 2. Is the sum of [R&D achievements] and [Prospects for commercialization] three or more points ? yes Go to 3. No Not enough 3. Is the sum of [R&D achievements] and [Prospects for commercialization] four or more points ? Excellent No Good - R&D Evaluation Methodology (Rating Method) 1. Each evaluator grades each evaluation item (A, B, C or D). A: Excellent, B: Good, C: Acceptable, D: Not acceptable 2. An average of the rating is used as an indicator. A=3, B=2, C=1, D=0 【example】 Categorize projects based on scores : 1. Are the respective categories one or more points ? yes

19. Ex-post Evaluations Example : Score Results for Ex-post Evaluation 3.0 Excellent X+Y=4 2.5 11 Good 8 4 6 2.0 X+Y=3 2 5 7 7 15 Prospects for commercialization (Y) 1.5 14 9 Not enough 13 10 1 1.0 3 0.5 12 0.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 R&D achievements (X)

20. Follow-up Monitoring & Evaluation Post-Project Activities of Companies Ex-Post Activities NEDO’s R&D Project (3-5 years) Commercialization R&D Tracked with the Follow-up Monitoring Target of follow-up monitoring is all of the companies that participated in NEDO funded projects

21. Objective Hypothesis : Are projects that scored high at ex-post evaluation also showing good results at follow-up monitoring ? Result of ex-post evaluation [Excellent] [Good] [Not enough] + Follow-up monitoring Study of Relation between Ex-post Evaluation and Follow-up Monitoring

22. Data usedfor this study Follow-up monitoring (Fiscal years) 2003 2004 2005 2006 Ex-Post evaluation (Fiscal years) 2002 4 1 2 3 2003 3 1 2 : Target of this study : 1,2,3,4 The number of passed follow-up monitoring years The number of projects and companies : 51 projects and 333 companies

23. Excellent Good Not enough 3.0 Excellent 2.5 Good 2.0 Prospects for Commercialization (Y) 1.5 Not enough 1.0 0.5 0 0 0.5 1.0 1.5 2.0 2.5 3.0 R&D achievements (X) Fig.1Score Results for Ex-post Evaluation （Projects ending in fiscal years 2001 and 2002） The number of projects Excellent : 22 projects Good : 19 projects Not enough : 10 projects

24. Excellent Good Not enough Excellent : 142 companies Good : 145 companies Not enough : 46 companies Result 100 80 66.0 66.9 65.5 65.5 66.0 64.1 58.7 58.8 58.0 60 Continuity R&D ( % ) 40 20 0 1st year 2nd year 3rd year The number of follow-up monitoring years Fig.2 Variation in persistence rate of R&D after the projects The number of companies

25. Excellent Good Not enough 20 17.3 18 15.9 14.8 16 13.1 13.0 14 12.0 11.8 11.8 11.8 12 The rate of commercialization ( % ) 10 8 6 4 2 0 1st year 2nd year 3rd year The number of follow-up monitoring years Fig.3 Variation in the rate of commercialization NEDO’s definition of “Commercialization” A company engages in marketing and sales generating activities.

26. Conclusion The R&Dcontinuity rates for three years after the “excellent", “good", and “not enough" projects had ended were compared, but a big difference was not seen. The evaluation result shows that even a project that is rated as “not enough" had some impacts. There was no significant difference between “good" and “not enough" projects, but the rate of commercialization for “excellent” projects increased during three years of the evaluation period. From the follow-up monitoring results, we find that projects rates “excellent” in the ex-post evaluation have higher rate of commercialization. However, we need a longer term data to understand the of the applicability ex-post evaluation.

27. Thank you for your kind attention. To be continued….

28. Introduction toFollow-up Monitoring and Evaluations

29. Objectives of Follow-up Monitoring and Evaluations • Objectives of this new initiative are: • 　・To understand in what ways the outcomes of R&D provides benefits to the public. • ・To draw lessons from R&D project outcomes to improve NEDO’s management and planning functions. Ex-post Activities of Private Companies Ex-post Activities NEDO’s R&D Project (3-5 years) Commercialization R&D Tracked by the Follow-up Monitoring

30. Process of Follow-up Monitoring and Evaluations Preliminary Preparation Identify and review items necessary for survey and evaluation 【Step1】 NEDO develops Follow-up Charts using information obtained at the interviews. Simplified Monitoring Grasp the current situation of an organization’s activities 【Step2】 Detailed Monitoring Detailed monitoring based on simplified monitoring 【Step3】 Evaluation Evaluation based on the above information

31. Study of Important Management Factors Based on Follow-Up Monitoring Data (Part II) Kazuaki Komoto New Energy and Industrial Technology Development Organization

32. What is the “Follow-Up Chart”? • Through interviews with companies, NEDO tries to identify specific management factors that led to R&D success and the practical application of such achievements. • The “Follow-Up Chart” concept was developed to visualize such factors in chronological order. • NEDO endeavors to verify the effectiveness of this method as well as to draw from lessons learned and make recommendations to improve its project management.

33. Q. When the project started, did you think you would achieve commercialization? A. Well, no. We thought this technology was very difficult. A. In 200X, we collaborated with Dr. YY. After that, we achieved a breakthrough. In 200X, we were able to commercialize our technology. Q. When… ?, What happen…?, How… ?, Who… ? How to make the “Follow-Up Chart” Potential for practical use Planning Project Post-project Interviewer (NEDO) Respondent (Project Participant) Follow-Up chart

34. In research phase, good collaboration among private companies. Case Study ＜Device development by collaborating with upstream and downstream industrial technology＞ Potential for practical use Achieved world record performance High Further technical problems requiring solutions before they can be practically applied Network of personal contacts Increased opportunities for business Achieved three times as high as performance by new method. ->Appropriate evaluation based on the R&D results. Post Project Evaluation In the development phase, upstream and downstream industry work closely together. legend Excellent collaboration between industry and universities Plus element Minus element In the practical phase, it can be difficult to maintain effective collaboration among companies. Intermediate evaluation Accelerated R&D efforts by setting very challenging target Low NEDO project encouraged newcomers to collaborate with universities, allocate new budget for R&D, etc. planning 1st yr. 2nd yr. 3rd yr. 4th yr. 5th yr. Post project Evaluation present

35. Type of “Follow-Up Charts” (commercialization achieved) After interviews with 49 companies that realized commercialization, each company was categorized as either type A or type B. Type A Type B Large Large 13 / 49 33 / 49 Mid Mid Potential for practical use Potential for practical use Small Small Time Project Time Project • Overly optimistic and unable to solve technical issues in the early stages • Breakthroughs in R&D during the latter half of a project • Define technical issues at the time of project planning and then gradually overcome the issues • R&D is based on a clear vision of social and user needs

36. Type of “Follow-Up Charts” (R&D discontinued after project completion) After interviews with 26 companies that discontinued R&D and did not realize commercialization, each company was categorized as either type C or type D: Type D Type C Large Large 8 / 26 15 / 26 Mid Mid Potential for practical use Potential for practical use Small Small Time Project Project Time • Steadily overcame technical problems. • No commercial (market) needs identified and no cost-competitiveness achieved. • Too optimistic to solve technical problems. • Cannot achieve breakthrough in R&D during the latter half of a project.

37. Type B Type A 1) 2) 5) 3) 4) Lessons from Case Studies: Type A and Type B • Choose project participants (entrusted companies/research institutions) who have good technical capability not only for NEDO’s project theme but also for related technology areas. • Use themidterm evaluation as an opportunity to direct the mindset of participants toward the practical applications and commercialization of research results. • Call for the participation of a key person in each company/research organization who can integrate the R&D and business divisions or oversee the entire project process from R&D to commercialization. • Make the responsibility and mission of each participant clear, and find a project leader who has strong leadership skills. • Facilitate collaboration between different types of businesses and user companies. Standard development should be done through collaboration among companies in the same industry.

38. Lessons from Case Studies: Type C Main cause of a downturn in Type C <Reference :Result of Questionnaire survey> The ratio of “Serving company purpose” • Companies that commercialized: 79% • Companies that discontinued projects: 44% Companies that have achieved commercialization tend to be motivated by serving company purpose • Change in company situation • 　・Dissolution or reorganization of departments • 　・Increased investment • Inability to adapt to a new social or economic situation • 　・Bursting of bubble in the information • technology sector • 　・Business environment deterioration related to • wasteful management • 　・Competitor takes leadership position in • building mass production sector • Technology was too advanced for practical application • ･New MEMS is too high spec for application • ･Cost reduction problems remain • ･Application of technology is far into future <Project Management Issues> ･ For a company to commit to project results. ･ User and application market trends need to be researched. ･ A cost target needs to be set (practical use project). ･ Technological superiority relative to competitive technology should be evaluated.(reflected in midterm evaluation standard).

39. Lessons Drawn from Case Studies : Type D <Project management issues> ･ Define technical problems at time of planning project (basic research project). ･ Attain higher possibility of achieving project objectives (reflected in midterm evaluation standard). ･ Selection and concentration of project theme and participants is important. Main cause of a downturn in Type D • Technical problems cannot be solved. • ・Problems recognized after implementation of projects

40. How to get feed back on NEDO management Management factors identified in follow-up monitoring are reflected in NEDO “Project Management Guidelines” Chapter II. Checklists for each phase Chapter III. Cases Checkpoint Description Case 成功に導くマネジメント例 教訓とすべきマネジメント例 優れたプロジェクト担当者の実践例 Guideline contents are confidential

41. Conclusion 1. By using a follow-up chart, we can visualize a project lifecycle , including various events which have occurred. 2. Follow-up chart can be used to derive lessons to improve R&D management. We can categorize follow-up charts into similar patterns and derive common lessons based on those patterns. 3. “Type C” companies demonstrate the problem where advanced technologies are developed but can not be taken to the market. It is a serious problem that we cannot completelyrecognize the market potential, or lack of market potential, of a technology during a project’s R&D phase. We have to consider seriously how to avoid “Type C” companies.

42. Outcome Survey Mid- and long term outcomes: Outcome surveys of the economic and social impacts of these technologies are conducted to improve NEDO’s accountability. 2.Materials 4.Energy conservation 1.Electronics Fine Ceramics Advanced Industrial Furnace Semiconductor Production Equipment Further details will be discussed later 8.Environment 7.Electricity storage Lithium Rechargeable Battery Incineration and Recycle System of Chlorofluorocarbon 3.Machinary Microelectromechanical System 5.New energy Photovoltaics 6.Fuel cells Fuel Cell

43. An Approach to Understanding Outcomes of National R&D in Energy Conservation Projects (Part III) Kazuaki Komoto New Energy and Industrial Technology Development Organization

44. What is a furnace? A furnace is used for purification of materials, heating processes, etc. There are 40,000 furnaces. The market size was 226 billion yen in 2005 in Japan. The energy consumed by furnaces is large. (They account for about 18% of final energy consumption in Japan.) Advanced Industrial Furnace Development Project １． Technical targets （１）　Energy conservation (more than 30%) – related to CO2 reduction （２）　Reduction of NOx （３）　System downsizing (more than 20%) ２． Term and budget （１）　Term： FY1993 – 1999 （7 years） （２）　Budget： About 8 billion yen ３． Project participants Associations, furnace makers, steel makers, etc. （14 organizations）

45. Output of the Project High-temperature air combustion technology has been commercialized by combining with an alternative regenerative burner. New combustion region (Energy conservation: good NOx: good) High-temperature air combustion technology (Output) Concentration of Oxygen in combustion air High Low Advanced Industrial Furnace Energy conservation: good NOx: bad High-temperature air combustion region Regenerative burner High-temperature combustion region Regenerator Fuel Furnes Room temp. air Preheated air Temperature of preheated air Combustion exhaust Exhaust Unstable combustion region Valve Valve Regenerator is heated by exhaust Change valve at regular time interval Alternative combustion Energy conservation: bad NOx: good High-temperature combustion region Normal combustion region High-temperature air combustion region Normal combustion region When valve is changed, the air is conducted in the burner.

46. Outcome of the project Application to steel making process Blast furnace Steel converter Ironstone Heating furnace Slab / Bloom / Billet Advanced Industrial Furnace End products ・Vehicles ・Water pipes, Gas pipes ・Household appliances ・Pipelines ・Business machines ・Ships, Bridges, Buildings Heavy plate Steel pipe Rolling H-shape steel Rolled steel Parts Heat-treating Heating furnace Forging Advanced Industrial Furnace Advanced Industrial Furnace

47. Market (total number of installed advanced industrial furnaces) The installation of advanced industrial furnaces has steadily increased in recent years. * Result of survey questionnaire answered by 79 companies. 668 furnaces

48. Market (Production volume of advanced industrial furnace) The production volume of advanced industrial furnaces has been increasing since the field test project was carried out. * Result of survey questionnaire answered by 79 members. 15 \12 billion Introduction and dissemination activities Field test project 10 Technical development project Production volume (\ billion) 5 0 92 93 94 95 96 97 98 99 00 01 02 03 04 05 Year

49. Value to Energy Conservation and CO2 reduction The “Advanced Industrial Furnace” contributes to mitigation of energy and environmental issues. Amount of energy conservation (in 2005): 335,000 kiloliters-oil/year 【Reference】 ・Photovoltaic: 277,000kL-oil/year (2004) ・Wind power: 378,000kL-oil/year (2004) 【Note】 168 million drums ・・・ (Drum: 200 liters) Amount of CO2 reduction (in 2005): 837,000t-CO2/year 【Note】 This is equal to the amount of CO2 that a 2,344 km2 forest can absorb in a year. 【Reference】 ・Tokyo metropolitan area: 2,187 km2 ・・・ （Amount of CO2 absorption in a year: 97.4 t-C/km2）

50. Case where project was carried out (Actual Case) Effect of project How do we estimate this case? Entire Case where project was not carried out (Hypothetical Case) Quantification of Project Outcomes Issues in quantification of outcomes • When we measure outcomes, we need to compare the outcome of the project to the non-project case. • It’s difficult to estimate outcomes when no project has been carried out. Outcome (production volume, etc.) Project Time