Introduction of Research and Development Project Evaluation System at NEDO

1. Introduction of Research and Development Project Evaluation System at NEDO 28 October 2005CES/AEA Conference in the Sheraton Centre, Toronto Momoko OKADA New Energy and Industrial Technology Development Organization(NEDO)

2. -CONTENTS- • ABOUT NEDO • R&D EVALUATION SYSTEM AT NEDO -INTERMEDIATE & EX-POST EVALUATIONS -FOLLOW-UP MONITORING& EVALUATIONS • CLOSING

3. -ABOUT NEDO- • Establishment -Established in October 1980 as a semi-governmental organization under the supervision of METI. - Original activities : 　　 ・national R&D projects on new energy -Current activities : ・Support of industrial technology R&D (in 1988), ・Support of environmental technology R&D (in 1990), and ・Introduction and promotion of new energy and energy conservation technology( in1993). -NEDO became an Incorporated Administrative Agency as of October 2003 in the process of recent government reform.

4. -ABOUT NEDO- • Mission -NEDO’s current missions are: 　 ・Promote new energy and energy conservation technologies through R&D and subsidy programs, ・Conduct and support industrial technology R&D, and ・Enhance international cooperation concerning energy and other technologies.

5. Machinery System Technology Development Dept. Environment Technology Development Dept. Fuel Cells & Hydrogen Technology Development Dept. New Energy Technology Development Dept. Energy Conservation Technology Development Dept. Coal Projects Dept., Coal Mine Subsidence Dept. -ABOUT NEDO- • Organization Staff : approximately 1,000 Budget : US$2.1 billion (FY2005) Headquarters Chairman, President & Executive Directors Auditor General Management Depts. General Affairs; Accounting; Assets Management; Inspection and Operational Management; and Information and Systems. Electronic & Information Technology Development Dept. Nanotechnology & Materials Technology Development Dept. Policy Planning and Coordination Dept Biotechnology & Medical Technology Development Dept. R&D Project Evaluation Dept. R&D Promotion Dept. Energy and Environment Policy Dept. International Projects Dept.

6. -ABOUT NEDO- • R&D support -R&D support is NEDO’s core activity. ・58% of the total budget (US$1.2 billion) is allocated for R&D activities in FY 2005. ・Number of NEDO’s R&D projects in FY2005 is 174. NEDO’s R&D Technology Areas • - Electronics and information - Environment • Nanotechnology and materials - Fuel cells and hydrogen • Biotechnology and medical technology - New energy • Machinery systems - Energy conservation

7. -ABOUT NEDO- • Challenge for NEDO -Need more effective and efficient operation through: ・Improved services, and ・Flexible resource management

8. -R&D EVALUATION AT NEDO- • R&D Project Management Cycle Policy Planning and Coordination Dept. PLAN Board of Executive Directors Technology Development Depts. SEE DO R&D Project Evaluation Dept.

9. -R&D EVALUATION AT NEDO- • Purposes of R&D Project Evaluations as an important part of the R&D project management cycle. -To reflect economic and social needs on R&D activities, -To implement R&D projects more effectively and efficiently, and -To improve project management system

10. -R&D EVALUATION AT NEDO- • 5 Principles of Evaluation at NEDO Transparent -Public disclosure of evaluation results, methods, and reflections Explicit -Encourage discussions Effective -Adopt an evaluation method for self- improvement of management Neutral -External evaluators Efficient -Simple evaluation procedures

11. Ex-ante evaluation (2004-) At R&D theme selection Intermediate evaluation (2001-) In the middle of project term Ex-post evaluation (2001-) 1 year following project completion Follow-up monitoring & evaluation (2004-) For 5 years after project completion ▼ ▼ ▼ ▼ Project implementation Post-project activity -R&D EVALUATION AT NEDO- • R&D Project Evaluations

12. -R&D EVALUATION AT NEDO- • Intermediate & Ex-post Evaluations -In all procedures, 5 principles are followed -Each project is assessed and evaluated from the following areas : ・purpose and strategy, ・project management, ・R&D achievements, and ・prospect for practical applications and other impacts

13. -R&D EVALUATION AT NEDO- • Intermediate & Ex-post Evaluations - R&D Evaluation Methodology (Rating Method) 1. Each evaluator grades (A, B, C or D) for each evaluation item. A: Excellent, B: Good, C: Acceptable, D: Not acceptable 2. An average of rating is used as an indicator. A=3, B=2, C=1, D=0 【example】 Items for evaluation average ratings 1. Purpose and strategy 2.3 A A B B B B 2. Project management 1.8 A B C B C B 3. R&D achievement 1.5 B C C C B B 4. Prospect for practical applications and other impacts 1.2 B B D C C C

14. -R&D EVALUATION AT NEDO- Score Results for Ex-post Evaluation in FY2004 3.0 X+Y=4 Excellent 2.5 20 3,4 8,30 14 11 15 X+Y=3 6 5,10,28 2.0 22 1 2 Good 3.0 25 13,24 12 27 26 Prospect for practical applications （Ｙ） 18 9 17 29 7 1.5 9 2.5 21 23 1.0 2.0 16 0.5 1.5 0.0 1.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 R&D achievement (X)

15. SubcommitteeD SubcommitteeB SubcommitteeC SubcommitteeA Promoting Department Evaluation Secretariat Promotion Department Project Participants -R&D EVALUATION AT NEDO- • Intermediate & Ex-post Evaluations Public (citizens) Releasing of evaluation results to the public NEDO Reflection of evaluation results in projects, etc. Chairman (13 persons） Evaluation report R&D Project Evaluation Committee Deliberation and approval of the evaluation report (draft) (4-9 persons） Preparation of evaluation report (draft) Explanation of the project

16. -R&D EVALUATION AT NEDO- • Intermediate & Ex-post Evaluations -In response to evaluation results: • Lessons learned from ex-post evaluations may be reflected upon other on-going or new projects. • Intermediate evaluations may result in modification or termination of projects. • Responses to Intermediate Evaluation Results (start) *some items are overlapping: all items added up do not match the total

17. -R&D EVALUATION AT NEDO- • Follow-up Monitoring & Evaluations (since 2004) -Objectives of this new initiative are: ・To understand in what ways specifically the outcome of R&D provides benefits to the public ・To reflect the outcome of R&D in improvement of the management and planning function at NEDO

18. Post-Project Activities of Private Companies Post-Project Activities NEDO’s R&D Project (3-5 years) Practical Applications Commercialization R&D Tracked with the Follow-up Monitoring -R&D EVALUATION AT NEDO- • Follow-up Monitoring & Evaluations

19. -R&D EVALUATION AT NEDO- • Follow-up Monitoring & Evaluations -NEDO initiatives as of October 2005: Subjects: 56 R&D projects , 642 entities ・These projects subject to follow-up monitoring & evaluations have received ex-post evaluation in FY2002-2003. ・These entities include private companies, universities, research institutes, and public institutions.

20. -R&D EVALUATION AT NEDO- • Follow-up Monitoring & Evaluations • Findings: -64% of the above entities are currently conducting post-project activities. -22% of the entities have reached a stage of practical applications or commercialization of R&D achievements. *Detailed information is being collected from 51 entities. • Utilization: -Make suggestions for improvement -Document key points to R&D management as a guideline

21. CLOSING ・Evaluations will play an increasingly important role in: -Improving performance of NEDO as an Incorporated Administrative Agency, and -Facilitating effective and efficient implementation of R&D project managements.

22. CLOSING • NEDO intends to • ・Review the current evaluation system • ・Make appropriate modifications • ・Propose new initiatives

23. Thank you for your kind attention. To be continued…