1 / 30

Meta-evaluation of the Performance Evaluation System of Public Research Institutes in Korea

AEA Evaluation 2009 Conference Nov. 11-14, Orlando, Florida. Meta-evaluation of the Performance Evaluation System of Public Research Institutes in Korea. Chan Goo Yi (Pukyong National University, Korea ; changoo@pknu.ac.kr ) Jang Jae Lee

inez
Download Presentation

Meta-evaluation of the Performance Evaluation System of Public Research Institutes in Korea

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AEA Evaluation 2009 Conference Nov. 11-14, Orlando, Florida Meta-evaluation of the Performance Evaluation System of Public Research Institutes in Korea Chan Goo Yi (Pukyong National University, Korea ; changoo@pknu.ac.kr) Jang Jae Lee (Korea Institute of Science and Technology Evaluation and Planning (KISTEP) ; jjlee@kistep.re.kr) Yong Soo Hwang (Korea Science and Technology Policy Institute (STEPI); yshwang@stepi.re.kr)

  2. Contents 1. Introduction 2. Research Method and Framework 3. Korean Research Institute Evaluation 4. Current Practise & Meta-evaluation 5. Discussion for Future Development 6. Conclusion 7. References

  3. 1. Introduction  Background - In Korea, evaluation system of research institutes introduced in 1999, and transferred into the performance evaluation system in 2005 - Arguments among various stakeholders such as CEO, researchers and evaluation panel, whether evaluation can contribute • the quality enhancement of R&D results • the development of management system

  4. 1. Introduction(con.)  Research Purpose - To meta-evaluate (1) the rationality of evaluation system itself and (2) the appropriateness of its current practise - To discuss policy alternatives for development of the evaluation system itself and its implementation

  5. 2. Research Method & Framework  Research Method - In-depth interview with 109 stakeholders • conducted between March and May 2008 • 99 internal stakeholders : CEOs(10), managers in administrative dept.(27), principal investigators(30), researchers(32) • 10 external stakeholders : evaluation panels from university(4), industry(3) and public research institute(3) - Meta- evaluation approach : Evaluation of evaluation system and practical process

  6. 2. Research Method & Framework(con.)  Research Framework : Components of In-depth Interview & Meta-evaluation Implementation(4) Paradigm(2) PurposeObject Utilization(2) Impact Type Panel Interval Method Indicator

  7. 3. Korean Research Institute Evaluation  Brief Historyof Institute Evaluation System - Introduction period (1999-2002) • similar system operated among research councils • research achievements < management achievements - Diversification period (2002-2005) • improved representing characteristics of individual member research institutes • research achievements ≒ management achievements - Development period (2006-current) • transferred performance evaluation system • focusing rather outcome or impact than output • research achievements > management achievements

  8. 3. Korean Research Institute Evaluation(con.)  R&D Governance in Korea : 3 Research Councils System - Korea Research Council for Fundamental S&T (KRCF) • 13 member research institute • under the Ministry of Education, Science & Technology - Korea Research Council for Industrial S&T (ISTK) • 13 member research institutes • under the Ministry of Knowledge Economy - National Research Councils for Economic, Humanities and Social Sciences (NRCS) • 23 member research institutes • under the Office of the Prime Minister

  9. 3. Korean Research Institute Evaluation(con.)  Framework of Current Evaluation System

  10. 4. Current Practise & Meta-evaluation (1)Evaluation Paradigm Evaluation Purpose - Formal : future development strategies, research performance enhancement, accountability, program/management improvement, knowledge transfer etc. - Actual : R&D program/project development, internal management system improvement (Meta-evaluation) - Incompatible between formal purposes and actual - Focused rather short-perspective evaluation purposes

  11. 4. Current Practise & Meta-evalu.(con.) Evaluation Object - Research Achieve.(70%)vs.Management A.(30%) •Each field divided into sub parts and items - Covering both ‘basic R&D program’(grant fund) and ‘national R&D program’(competition fund) (Meta-evaluation) - Too many evaluation objects • Not differentiating among evaluation objects - Absent of essential objects for core evaluation purposes such as ‘future development strategy’ - Lacks of consensus for core evaluation objects among stake-holders

  12. 4. Current Practise & Meta-evalu.(con.) (2)Evaluation Implementation Evaluation Panel - Panel from university, research institute and industry, & comprising all domestic experts • Research : individual panel for each institute • Management : common panel for all institutes (Meta-evaluation) - Panel members’ professionalism limited • Professionals in sub performance goals of R&D project rather than peer reviewer or upper performance goal in R&D program - A few lacks of considering international excellency

  13. 4. Current Practise & Meta-evalu.(con.) Evaluation Interval - Until 2007 : evaluating all institutes every year - From 2008 : separating research achievement and management one • Research A. : 3 years ; Management A. : 1 year (Meta-evaluation) - Too often evaluated and burden to institutions • Main factor negative affecting other components such as evaluation purpose, object, utilization - Resulted in more focusing the visible and short term outputs rather than long term and comprehensive outcomes or impact

  14. 4. Current Practise & Meta-evalu.(con.) Evaluation Method - External panel conducts full evaluation process - Evaluating 4/5 member institutes as one group in a comparative perspective with others - Combination of the ‘review of performance report’ submitted by each institute and the ‘site visit’ for four or five hours in individual institute (Meta-evaluation) - Focused on literature review rather than site visit - Evaluation in a comparative/relative perspective not an absolute one, in particular for the research performance

  15. 4. Current Practise & Meta-evalu.(con.) Evaluation Indicator : Focused on Research - Each Program(50%) vs. Comprehensive R&D(20%) •Performance goals -> Objects -> Indicators - Each institute suggests their own goals/indicators • Evaluating performance level targeted in advance (Meta-evaluation) - Disconnection of performance goal and core R&D activities - Indicator pool limited for comprehensive R&D performance goal - Indicators more fit for research program rather than development one

  16. 4. Current Practise & Meta-evalu.(con.) (3)Evaluation Utilization Evaluation Impact - Evaluation findings directly fed back to the basic R&D(grant fund), but indirectly did to national R&D(competition fund) - Feed back to internal management system (Meta-evaluation) - Evaluation findings more effecting management system rather than R&D management process - Lacks of consensus of impact among stakeholders • Insiders : low, Outsiders : relatively high

  17. 4. Current Practise & Meta-evalu.(con.) Type of Evaluation Utilization - Formal : future development strategy, R&D/ management improvement, R&D prioritization and budget allocation, best practise, CEO’s annual pay adjustment, policy suggestion - Actual : budget reallocation of basic R&D, adjustment of CEO’s annual payment, best practise (Meta-evaluation) - Limited and confined evaluation utilization - Instrumental utilization is further actual type than conceptual one

  18. 5. Discussion for Future Development  Outline of Discussions for Developments - Institutional Approach (IA) • Development/amendment of evaluation system itself and related systems at the level of research councils or the government • Long-term and institution-based perspective - Operational Approach (OA) • Improvement of evaluation practise and process under the current system • short-term and operation-based perspective

  19. 5. Discussion for Future Development(con.) (1)Evaluation Paradigm  Evaluation Purpose - Need to transfer from internal responsibility to external one (OA) • Because current system has more contributed internal responsibility/ management rather than external one - More focusing responsibility for external stakeholders (OA) • In particular, responsibility for citizen

  20. 5. Discussion for Future Development(con.)  Evaluation Object - Rearrangement of current evaluation object (IA) • Whether competitive national R&D program be included or not? - Re-setting the evaluation objects in the longer perspective, such as; (IA) • Long-term vision/strategy of the institution • Future potentials and R&D infrastructures • Risk management in related public sector

  21. 5. Discussion for Future Development(con.) (2)Evaluation Implementation Evaluation Panel - Setting up individual/independent panel for each institution, from single panel for all (IA) - Enhancement of professionalism of panel (OA) • Recruiting more field experts like as industries • Extension of job term from 2 to 4/5 years • Career management system of panel members - Adoption of international experts panel (OA)

  22. 5. Discussion for Future Development(con.) Evaluation Interval - Extension of evaluation interval from 1 year to 3 to 5 years (IA) • Linkage with CEO’s term in office • Same or different evaluation interval between research results and management results • Consideration of characteristics of research fields; such as emerging technology vs. long-term basic science 22

  23. 5. Discussion for Future Development(con.) Evaluation Method - Enforcement of evaluation method in the way of absolute perspective (IA) • In particular, evaluation of research results - Focusing in-depth review in research lab (IA) • Extending evaluation period to 3 to 4 days • Reviewing first-hand materials(research note) • Interview and discuss with researchers - Introduction of cross-cutting review • Among related institutions/organizations in public sector (IA) 23

  24. 5. Discussion for Future Development(con.) Evaluation Indicator - Improvement of method of establishing performance goals/indicators (OA) • Closerconnectivity between performance goal/indicators and core R&D activities • More changeable and creative goals/indicators - High linkage between performance goals and internal performance management system (OA) • For example, BSC, MBO, ISO 9001, KM etc - Increase of indicator pools for comprehensive performance goals (OA)

  25. 5. Discussion for Future Development(con.) (3)Evaluation Utilization Evaluation Impact - Enforcement of feed back system of evaluation findings to R&D management (OA) • Highcollaboration with related other agencies for national R&D management - Extension of scope and target group of evaluation utilization (OA) • From top manager to all employees

  26. 5. Discussion for Future Development(con.) Type of Evaluation Utilization - Activation of long-term and conceptual evaluation utilization (OA) • Setting up vision/mission • PlanningR&D strategy • Disseminating the best practise • Producing policy information/knowledge etc - Informing the multiple type of evaluation utilization to all stakeholders (OA) •In particular, in-site researchers

  27. 6. Conclusion  Summary of the research - In basic, both internal and external stakeholders consider the current system somewhat useful for R&D management and organizational management - They also suggest policy alternatives for the development of certain components of evaluation system and its current practise • Some are the system itself, others are the common limits of evaluation system of public sector in general in Korea

  28. 6. Conclusion (con.)  Implication - Policy alternatives for developments of evaluation system for research institutes in terms of system itself and current practise - Policy knowledge/ information for analysis and re-establishment of the governance of public research institutes

  29. 6. Conclusion (con.)  Limitations andFurther Works - Discussion of policy alternatives a little bit lacks specific and detailed matters in some individual components • Scope and depth of policy alternatives in certain components is too broad - Need for more detailed action plan in each evaluation component, based on the findings of this meta-evaluation/research

  30. 7. References  FurtherInformation for Korean S&T and Evaluation System - National Science and Technology Council (http://www.nstc.go.kr) - Ministry of Education and Science and Technology (http://www.mest.go.kr) - Korea Research Council for Fundamental S&T (http://www.krcf.re.kr) - Korea Research Council for Industrial S&T (http://www.istk.re.kr) - Korea Institute of S&T Evaluation and Planning (http://www.kistep.re.kr) 30

More Related