New Direction of National R&D Evaluation System in Korea

1. EVALUATION 2011 @ Anaheim, CA, U.S.A. New Direction of National R&D Evaluation System in Korea 2011. 11. 3 Changwhan Ma Director of Performance Policy Division

2. Contents 13 slides Contents • Introduction • Background • Initiative Focus • Conclusion 1 3 5 13

3. Contents • Introduction • Background • Initiative Focus • Conclusion 1 3 5 13

4. R&DProgram Management Process 1 Policy (Planning) Feasibility Study (Ex-ante) Program (Planning) Evaluation strategy & Data collection R&D Budget Survey/Analysis Evaluation Implementation (Programs/Projects) Self → Meta In-depth Recommendation Feedback Evaluation process Utilization process

5. National R&DEvaluation System in Korea Efficiency/Effectiveness Of R&D Investment ↑ (Guidelines) In-depth Eval Results (Grades, Recommendations) Program Evaluation Ex-post Self Eval. Meta Eval Ex-post/post Follow-up Survey Follow-up Eval GRI Evaluation Management & Research Self Eval. Meta Eval Report to NSTC Ministries NSTC/(KISTEP) Feedback 2 Results (Budget Size, Recommendation) Feasibility Study Feasibility Study Ex-ante Program Planning NSTC/MOSF/(KISTEP) Preliminary Study Ex-post *NSTC: National Science & Technology Commission

6. Contents • Introduction • Background • Initiative Focus • Conclusion 1 3 5 13

7. Background of New Direction • Continuous increase of R&D investment  3rd on R&D investment / GDP (3.74%, 2010), 14.89 billion USD in 2011 • Quantitative Efficiency of R&D Performance has been continuously increased  (Publication and Patent) Near top 10 in the world • Qualitative Excellence is stalled around 30th in the world • S&T and Socioeconomic Competitiveness depend on the qualitative excellence of researchers and their performance • Therefore, evaluation system should meet the direction of S&T advancement • And new NSTC has recently been launched to respond to the recent change. 3

8. New NSTC 4 삭제 Constructing overall R&D System • Before After NSTC (Multi-departmental R&D program, etc.) Ministries, GRIs etc. Planning Suggestion about direction of budget allocation & Budget allocation / coordination of main National R&D program Suggestion about direction of Budget allocation NSTC Resource Allocation Allocation· Coordination· Formation Budget Formation MoSF NSTC MoSF ce Self evaluation Ministries Self evaluation Ministries Evaluation Meta& Specific evaluation Meta& Specific evaluation MoSF NSTC

9. Contents Contents • Introduction • Background • Initiative Focus • Conclusion 1 3 5 13

10. 1. Open Evaluation System 5 • To ensure variety of evaluation perspectives by increasing Openness of Performance Information and Various Opinions to the information • To promote expert review activity by establishing comprehensive expert community pool • To introduce on specific evaluation first and escalated to other evaluation types ① More Openness in Performance Information & Expert Participation ② Establishing Online Open Evaluation System • To share and communicate more by establishing user-friendly online open evaluation system (OOES) • To develop model and manual for OOES

11. 2. Mission-oriented GRI Evaluation System 6 • To introduce absolute rating system considering characteristics of each GRI (government-supported research institutes) • To expand the autonomyof GRI during evaluation process • To focus on accomplishment of GRI mission • To encourage to set challengeable objective to reach world class technology level ① Introducing Absolute Rating System ② Alleviating Burden on Evaluation • To reduce the portion of management evaluation gradually and integrated into performance evaluation (depends on revision of act on R&D performance evaluation)

12. 3. Rationality and Autonomy of Self Evaluation 7 • To ensure the quality of self evaluation by pre-check the appropriateness of performance objective and indicator • To operate official committee on reviewing performance indicator • To set up the evaluation schedule (or cycle) according to milestones of each programs by budget-spending ministry • To carry out meta evaluation in general and re-evaluation on specific programs ① Stringent Pre-check of Performance Plan ② More Autonomy on Self Evaluation

13. 4. Evaluation of Qualitative Excellence 8 • To carry out comprehensive evaluation considering scientific/technological and socioeconomic impact and sustainability • To encourage to set challengeable performance objective by using qualitative performance indicator ① Qualitative Evaluation considering Impact ② Guideline on R&D Performance Evaluation • To develop and disseminate R&D performance evaluation guideline for better understanding of qualitative evaluation and for strengthening evaluation capacity of ministries

14. 5. R&D Policy Evaluation and Cross-cutting Review 9 • To introduce R&D policy evaluation for better coordination of R&D programs and R&D environment • To expand the scope of specific evaluation from individual program to fields of technology and similar program groups • To ensure optimize and coordinate the delivery system the similar, overlapping programs ① Evaluation of R&D Policy and Issue ② Cross-cutting Review

15. 6. Strategic Performance Budgeting & Program Improvement • To set priority and allocate budget of the similar programs according to evaluation program selection and results • To carry out integrated performance budgeting by reviewing mid/long-term direction of R&D investment • To keep maintaining performance budgeting with the evaluation results ① Strategic Performance Budgeting ② Monitoring System of Performance Information • To establish the DB for evaluation history of the program including evaluation results, recommendation and management action plan, etc. • To ensure the continuous monitoring evaluation feedback efficiently 10

16. 7. Customized Evaluation 11 • To apply differentiated evaluation perspective according to types, size of the program • To develop evaluation model and checklists with common or specialized indicators ① Evaluation according to Program Type ② Milestone-based Performance Evaluation • To select the program after considering the cycle or time for producing key performance (creative period; 3 yr, 5 yr, exempted, etc.) • To encourage to carry out objective and comprehensive performance analysis for self monitoring

17. 8. Infrastructure and Network 12 • To establish DB for major performance and budget information as a program management platform • To serve as an active channel for the exchange of information on evaluation trend and various opinions • To carry out R&D performance evaluation capacity building ① Infrastructure of R&D Performance Evaluation ② Network of R&D Evaluation • To hold the evaluation forum regularly (global R&D evaluation network) • To carry out collaborative research on evaluation • To participate the conference on evaluation regularly

18. Contents • Introduction • Background • Initiative Focus • Conclusion 1 3 5 13

19. Future Direction 13 National R&D Program Open system Customized Mission … Qualitatively Excellent Researchers with Performance Strong (S&T → National) Competitiveness

20. Thank you!