Essential Factors for Effective and Efficient Project Management

Essential Factors for Effective and Efficient Project Management 28 October 2005CES/AEA Conference in the Sheraton Centre, Toronto Mutsumi Fujita, Eiji Okutani New Energy and Industrial Technology Development Organization(NEDO)

CONTENTS • INTRODUCTION • NEDO’S R&D EVALUATION • ANALYSIS OF EVALUATION RESULTS • OVERVIEW OF EVALUATION RESULTS • MANAGEMENT AND ACHIEVEMENT • HIGH SCORING PROJECTS AND LOW SCOREING PROJECTS ON MANAGEMENT • CONCLUSIONS

INTRODUCTION • NEDO has conducted intermediate and ex-post evaluations for about 140 projects since 2001. • Through R&D project evaluations, NEDO aims to: • Reflect economic and social needs to its R&D activities, • Execute more effective and efficient R&D projects, and • Improve its management system.

INTRODUCTION • Outline of Survey • Objective: to acquire effective and efficient project management methods through analyses of results of ex-post evaluation. • Methodology: to identify important management factors by analyzing evaluators’ comments and recommendations on high scoring projects in the ex-post evaluation. • Survey targets: 30 projects for which ex-post evaluation were conducted in FY 2004.

NEDO’S R&D EVALUATION • General procedures: • external evaluators, 4-9 members • Each project is assessed and evaluated based on the following four areas: • Purpose and strategy; • Project management; • R&D achievements; and • Prospect for practical application and other impacts. • Evaluators give comments and recommendations, and grade each project using a four-level scale (A-D).

NEDO’S R&D EVALUATION • Grading scale • Each project is given a grade by the evaluators. • A : Excellent, B : Good, C : Acceptable, D : Not acceptable • A grade point average is calculated using the following points. • A=3, B=2, C=1, D=0

ANALYSIS OF EVALUATION RESULTS • OVERVIEW OF EVALUATION RESULTS Table1. Average score of 30 ex-post evaluation results in FY 2004. Perfect score : 3

3.0 20 11 25 9 12 22 3 4 6,21 2.5 30 28 8 27 13 5,10 29 7 15,24 1 2.0 26 17 14 2 19 18 1.5 16 Score of “Achievement” 23 1.0 0.5 0.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Score of “Management” ANALYSIS OF EVALUATION RESULTS • MANAGEMENT AND ACHIVEMENT R2 = 0.57 Figure 1. The relation of “Management” to “Achievement”. Each spot stands for each project. • Good “Management” seems to be associated with good “Achievement”.

ANALYSIS OF EVALUATION RESULTS • HIGH SCORING PROJECTS AND LOW SCORING PROJECTS ON MANAGEMENT • Comparison Methods • Extract comments and recommendations regarding “Project Management” from 30 evaluated projects. • Comments and recommendations were classified according to the following criteria, and divided into the following: • Comments and recommendations which are common among high-scoring projects • Comments and recommendations which are common among low-scoring projects

ANALYSIS OF EVALUATION RESULTS • Evaluation Criteria for Project management (1) Validity of target A) Have the targets been determined based on the technological and market trends? B) Have concrete and clear goals to achieve those targets been set? C) Have appropriate indicators to measure attainment been established? (2) Validity of plan D) Are the schedule and budget appropriate for the attainment of goals? E) Are necessary elemental technologies in place for the attainment of goals? F) Are the relation and order of elemental technologies appropriate? G) For long-term and continuing projects, how will the obtained results and knowledge be utilized?

ANALYSIS OF EVALUATION RESULTS • Evaluation Criteria for Project management (cont.) (3) Validity of project formation H) Is the project formation appropriate to attain the goals? I) Have researchers been selected appropriately? J) If the management office is involved, does it play appropriate roles? K) Has the project leader been selected appropriately? Has an environment to ensure success been created? L) Is the formation adequate to urge collaboration and/or competition among the participants? M) Based on a scenario for practical applications, is there a system to request the results to stakeholders (target users of applications) and to request their involvement? (4) Response to change of situation N) has progress been constantly monitored and properly reflected in the plan? O) Can there be a timely and appropriate response to changes in the socioeconomic situation and political and technological trends? P) Is the policy for revising the plan consistent?

P O A N B M C L D K E J F G I H ANALYSIS OF EVALUATION RESULTS • What are comments and recommendations in all projects? Positive comments (N=100) • 50% of all projects received positive comments for criteria A or B or N. • A: Targets have been determined based on technological and market trends.(Number of the project with positive comments for A is 12) • B: Concrete and clear goals to achieve those targets have been set. (Number of the project with positive comments for B is 15) • N: Progress has been constantly monitored and properly reflected in the plan. (Number of the project with positive comments for N is 15) Figure 2. Positive comments and recommendations in 30 projects.

B P A O N C D M E F L G H I J K ANALYSIS OF EVALUATION RESULTS • What are comments and recommendations in all projects? Negative comments (N=62) • 43% of all projects received negative comments for criterion L. • L: The project formation is adequate to urge collaboration and/or competition among the participants Figure 3. Negative comments and recommendations in 30 projects.

ANALYSIS OF EVALUATION RESULTS Table 2. Comments of high-scoring projects. • What comments or recommendations are common among high-scoring projects? Blue: 4 projects share same comments, Black: 3 projects share same comments.

ANALYSIS OF EVALUATION RESULTS • Necessary factors to lead to successful projects • A: Targets have been determined based on technological and market trends. • Sufficient preliminary surveys and studies have been conducted. • Project is not limited to self-serving purposes. • The vision is realistic. • B: Concrete and clear goals to achieve those targets have been set. • Setting quantifiable goals if possible. • N: Progress has been constantly monitored and properly reflected in the plan. • Checking the progress of study at every opportunity (e. g. Intermediate evaluation), and revising the plan as needed.

ANALYSIS OF EVALUATION RESULTS • Most important factors to make projects successful • K: The project leader has been selected appropriately and the environment ensuring successful activities has been created. • Choosing leaders with high technological and operational capabilities, as well as strong leadership skills to organize multiple study groups consisting of diverse companies and universities. • Leaders must be able to make decisions on issues involving finance, personnel, and research plans. • L: The project formation is adequate to urge collaboration and/or competition among the participants. • Sufficient discussion and communication among the participants. • Integrate results from multiple laboratories.

3.0 3.0 2.5 2.5 2.0 2.0 1.5 1.5 Score of “Achievement” Score of “Achievement” 1.0 1.0 0.5 0.5 0.0 0.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Score of “Management” Score of “Management” ANALYSIS OF EVALUATION RESULTS K)R2 = 0.74 (N=13) L)R2 = 0.75 (N=17) Figure 4. The relation between “Management” and “Achievement” in projects with comments regarding K or L.

ANALYSIS OF EVALUATION RESULTS Table 3. Comments of low-scoring projects. • What comments or recommendations are common among low-scoring projects? Blue: 4 projects share same comments, Black : 3 projects share same comments.

ANALYSIS OF EVALUATION RESULTS • Factors with no impact on success or failure • Positive comments for criterion N (Progress has been constantly monitored and properly reflected in the plan) are seen both in high and low-scoring projects. Thus, N seems not relevant to either success or failure of projects. • → Setting up a good plan at the beginning of a project is more important than revising it in the middle of a projects.

3.0 2.5 2.0 1.5 1.0 0.5 0.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 ANALYSIS OF EVALUATION RESULTS N) R2 = 0.38 (N=18) Score of “Achievement” Figure 5. The relation of “Management” to “Achievement” in projects with comments regarding N. Score of “Management”

CONCLUSIONS • Essential factors for effective and efficient project management... ○ The best project leader should be selected and the environment ensuring successful activities should be created. ○ Good collaboration and/or competition among the participants. ○ Concrete and clear goals to achieve those targets should be set based on survey studies and technological and market trends. ○ Setting up a good plan at the beginning of a project is more important than revising it in the middle of a projects

Thank you !

3.0 3.0 2.5 2.5 2.0 2.0 1.5 Prospect for practical use 1.5 Prospect for practical use 1.0 1.0 0.5 0.5 0.0 0.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Management Achievement y = 0.4172x + 1.0389 R2 = 0.2807 y = 0.5293x + 0.7246 R2 = 0.3463

Energy Environment 3.0 3.0 2.0 2.0 Prospect for practical use Prospect for practical use 1.0 1.0 Biotechnology and medical technology 0.0 0.0 0.0 1.0 2.0 3.0 0.0 1.0 2.0 3.0 Achievement 3.0 Achievement R2＝0.29 2.0 R2＝0.29 Prospect for practical use 1.0 Electronics and information Nanotechnology and materials 3.0 3.0 0.0 0.0 1.0 2.0 3.0 Achievement 2.0 2.0 Prospect for practical use Prospect for practical use R2＝0.61 1.0 1.0 0.0 0.0 0.0 1.0 2.0 3.0 0.0 1.0 2.0 3.0 Achievement Achievement R2＝0.56 R2＝0.66

K & L R2= 0.9645 3.0 11 3 4 2.5 2.0 1.5 Score of “Achievement” 16 y = 0.8433x + 0.3961 1.0 0.5 0.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 N = 6 Score of “Management”

not K R2 = 0.39 3.0 11 20 25 9 22 2.5 8 28 27 13 5,10 29 1 2.0 2 26 17 19 18 1.5 y = 0.5158x + 1.194 1.0 0.5 0.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 • not L R2= 0.56 3.0 11 12 6,21 22 4 3 2.5 30 8 27 13 5,10 7 15,24 2.0 14 18 1.5 16 Score of “Achievement” Score of “Achievement” 23 1.0 y = 0.694x + 0.7871 0.5 0.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Score of “Management” Score of “Management”

Essential Factors for Effective and Efficient Project Management