1 / 38

Research and Development Evaluation in Japan & MEXT

Research and Development Evaluation in Japan & MEXT. Naoko Okamura Director, Office of R&D Evaluation Science and Technology Policy Bureau Ministry of Education, Sports, Culture, Science and Technology (MEXT), Japan. Content. Ⅰ ． Overview 1. Administrative Structure for S&T in Japan

chaka
Download Presentation

Research and Development Evaluation in Japan & MEXT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Research and Development Evaluation in Japan & MEXT Naoko Okamura Director, Office of R&D Evaluation Science and Technology Policy Bureau Ministry of Education, Sports, Culture, Science and Technology (MEXT), Japan

  2. Content Ⅰ．Overview 1. Administrative Structure for S&T in Japan 2. National S&T Strategy 3. Structure of R&D Evaluation in Japan 4. History of R&D Evaluation in Japan Ⅱ．R&D Evaluation in MEXT 5. Summary of the MEXT Guideline 6. Current System of R&D Evaluation 7. Evaluation System in Competitive funds (Example with Special Coordination Funds for Promoting R&D 8. Evaluation System of RIKEN 9. Evaluation System of NIMS

  3. Ⅰ. Overview

  4. CSTP : Council for S&T Policy AEC : Atomic Energy Commission NSC : Nuclear Safety Commission NPA : National Police Agency MPHPT : Ministry of Public Management, Home Affairs, Posts and Telecommunications MOFA : Ministry of Foreign Affairs MOF : Ministry of Finance MEXT : Ministry of Education, Culture, Sports, S&T MHLW : Ministry of Health, Labour and Welfare MAFF : Ministry of Agriculture, Forestry and Fisheries METI : Ministry of Economy, Trade and Industry MLIT : Ministry of Land, Infrastructure and Transport MOE : Ministry of the Environment 1. Administrative Structure for S&T in Japan (Since Jan. 2001) Prime Minister Cabinet Office Minister of State (S&T Policy) CSTP ○Formulation of comprehensive strategy ○Policy for allocation of resources such as budget and human resources etc. AEC & NSC ○Designing, Planning, and coordinating Policies Presenting a Basic Policy, Comprehensive Coordination MEXT METI ○Formulation & promotion of concrete R&D plans ○Coordination among relevant ministries - Special Coordination Funds for Promoting S&T - Coordination on S&T international cooperation NPA MOFA MHLW MLIT MOE MPHPT MOF MAFF etc. University Public Corporation, Independent Administrative Institution, National Research Institute 4

  5. 2. National S&T Strategy 2-1 S&T Basic Plan for 2001-2005 • Three Basic Characteristics - Creating and utilizing scientific knowledge to contribute to the world - Developing international competitiveness and ability of sustainable development - Securing safety and quality of life • Basic Policies - Strategic Priority Setting - S&T System Reforms • Doubling competitive research funds, introduce indirect costs to enhance competitive atmosphere • Improving mobility of human resources in S&T • Increasing young researchers funds to exhibit their talent • Enforcing effective evaluation - Internationalization

  6. 3. Structure of R&D Evaluation in Japan National Level S&T Basic Plan General Guideline for R&D Evaluation • CSTP is promoting R&D evaluation as an integral part of S&T policy • Ministries and R&D Organizations carry out the major part ot evaluation activities Cabinet decision Decided by the Prime Minister Ministry Level METI Guideline MEXT Guideline Other Ministries Organization Level RIKEN Guideline JST Guideline NEDO Guideline NIAIT Guideline Program for Major Activities Program for Major Activities

  7. 4. History of R&D Evaluation in Japan • 1997: First National Guideline - Introduced R&D evaluation - Encouraged external evaluation - Aimed at efficient resource allocation • 2001: Revised National Guideline - To upgrade fairness and transparency - To strengthen link with budgeting - To establish resource base for evaluation • 2005: Revised National Guideline - To encourage researchers’ challenge for creativity - To develop evaluation skills to increase reliability - Evaluation results should be utilized more strictly MEXT has just established “Guideline for Evaluation of Research and Development in MEXT” in Sep.2005

  8. Ⅱ. R&D Evaluation in MEXT

  9. 5. Summary of the MEXT Guideline 5-1 Role - Sets out the basic approach for conducting evaluations on R&D which fall under MEXT’s jurisdiction. Internal divisions of MEXT should conduct evaluations based on this guideline while R&D institutions should refer to this guideline for conducting appropriate evaluations • Goals of the evaluation To promote • Research developing new frontier of S&T • Research contributing to the society/economy • Significance of the evaluation • Realizing open, flexible and competitive R&D environment • Allocating R&D resources efficiently and effectively • Obtaining public support through disclosure of R&D achievements 5-2 Basic Concept

  10. 5-3 Direction of Reform • Encouraging researchers’ challenge for creativity, promoting new frontier research • Recognize that former evaluation sometimes discourage researchers • Continuation of the excellent projects through evaluation • Announce the results of reflection • Ensuring the evaluation resources and improving evaluation system • Prepare necessary budget for evaluation process • Foster personnel for evaluation by training • Develop the National R&D database • Develop efficient evaluation • Eliminate too much procedure and too many data for evaluation

  11. 5-4 Common Principles • Setting the specific objective and aim of the evaluation • Selection and assignment of reviewers • Avoid conflict of interest • Timing of evaluation • Follow-up activity after the evaluation • Method of evaluation • Announcing the procedure, criteria, etc. to the object • Evaluating “quality” rather than “quantity” • Expanding objective indicators use taking consideration for their limit • Revealing criteria for evaluation • Securing flexibility and avoiding burden • Utilization of the results • Reflection to the distribution of resources

  12. 5-5 Category of Evaluation • Research and Development Policy • Program, system etc. • Research and Development Theme • Competitive fund • Project • Research and Development Institute • Independent Administrative Institutions • National Laboratories • Public Corporations • Universities • Achievement of researcher

  13. 6. Current System of R&D Evaluation (1) CST:Council for Science and Technology SAC:Space Activity Commission 13

  14. 6. Current System of R&D Evaluation (2) IAI:Independent Administrative Institute NUC:National University Corporation IAIEC: Independent Administrative Institute Evaluation Committee NUCEC: National University Corporation Evaluation Committee MPHPT:Ministry of Public Management ,Home Affairs,Posts and Telecommunications

  15. 7. Evaluation System in Competitive funds - Special Coordination Funds for Promoting R&D - POINTS ○SCF is fund fordeveloping new frontier of S&T and contributing to the society/economy. ○ Evaluation for various aspects . Scientific evaluation byPeer reviews Final Selection byExpert Panel (including industry researchers) ○ Working Groups are supported by Scientific stuff Program Officers,Program Director Ex-post Evaluation Screening (=Prior Evaluation） Hearingexamination (consultation) Selection of subjects 3rd year Acceptance of applications Documentary examination R&D ( 5 years ) by CST WGs, are established in the every fields and subjects are examined. Interim Evaluation Working groups (=expert panel) Peer reviews established in JST(=Japan S&T corporation)

  16. 8. Evaluation System of RIKEN8-1 Profile of RIKEN Basis for Establishment RIKEN Law (Independent Administrative Institution founded under Law No. 160, December 13, 2002) Date of Establishment October 1, 2003 Objective To conduct comprehensive research in science and technology (excluding humanities and social sciences) and to foster advances in scientific research and technological standards 2004 budget 2005 budget ¥83,956 million (of which ¥74,920 million is funded from the government subsidy) ¥86,796 million (of which ¥75,882 million is funded from the government subsidy) Employees as of end of 2004 2,835 full-time employees (of which 692 are executives and Mandatory Retirement Age Contract staff) President Ryoji Noyori, Dr. Eng. Principal operations (1) Comprehensive research and testing • basic science research; (b) promotion of accelerator scientific research; (c) promotion of bioresources businesses; (d) promotion of synchrotron radiation research; (e) promotion of integrated collaborative in research; (f) promotion of comprehensive brain science research; (g) comprehensive genome science research; (h) promotion of botanical science research; (i)comprehensive research in developmental biology; (j) promotion of SNP research; (k) promotion of comprehensive research into immunology and allergy science; and (l) promotion of strategic research (2) Dissemination and active utilization of research achievements Presentation of research reports, access to resources in biogenetics, establishment of intellectual property rights on scientific achievements, licensing of patents, etc. (3) Shared-use of facilities and equipment (4) Training of researchers and engineers and advancement of their qualities and skills (5) Promotion of shared use of Spring-8

  17. 8-2 Eight Features of RIKEN Operations 1. Longstanding reputation as a comprehensive research institute 5. Competitive and diversified research funding ・History of 88 years since 1917 ・Japan’s only comprehensive research institute of natural science ・Acquisition of external funding that is competitive and diverse ・Creation of competitive environment for funding inside the Institute 2. Broad range of research 6. Open research system ・Large number of fields (physics, chemistry, engineering, biology, medicine, etc.) ・Interdisciplinary research ・From basic science to applied research ・Joint research and broad-ranging personnel exchanges with universities, business enterprises, etc. ・Cooperation Program for graduate school with universities in Japan 7. International Institute 3. Versatile research organization adapted to the nature of research ・Large number of foreign researchers ・Three overseas research centers ・Cooperation with foreign research institutes (including NIH & Max Planck) ・Institution Laboratory (for emerging research area, compound research and integrated research areas; hub organization) ・Frontier Research System (new areas of research & target-oriented) ・Center system (centralized action in prioritized fields) ・Availability of research platform (bioresources, SPring-8, etc.) 8. Research evaluation on multiple stages 4. Promotion of alliances with industry and technology transfer ・First systematic evaluation by external sources, including foreign researchers ・Evaluation on research unit revels (by Centers, by laboratories, etc.) ・Evaluation of research themes & research performance ・Joint research, patent licensing, etc.

  18. RIKEN Headquarters Policy Planning Division, Public Relations Office, General Affairs Division, Personnel Division, Finance Division, Contract Management Division, Technology Transfer and Research Coordination Division, Facilities and Utilities Division, Safety Division, Auditing and Compliance Office, Internal Communications and Systems Support Office , Advanced Center for Computing and Communication, Center for Intellectual Property Strategies President Ryoji Noyori, Dr. Eng. Executive Directors Kenji Okuma Tsutomu Shibata Motohide Konaka Yoshiharu Doi, Dr.Eng. Kenji Takeda, Dr.Eng. • Wako Institute (Koji Kaya, Ph.D.) • Discovery Research Institute, Frontier Research System, Brain Science Institute. Initiative Research Unit, Special Laboratories, DRI/FRS Promotion Division, Brain Science Promotion Division • 　　　＜Overseas Centers＞ • RAL Laboratory, UK, BNL Research Center, US (Discovery Research Institute) • MIT Brain Science Research Center, US (Brain Science Institute) Research Priority Committee Tsukuba Institute (Tsutomu Shibata) Bioresource Center, Research Collaborative Group, Tsukuba Research Promotion Division, Tsukuba Safety Center Auditors Takashi Fujii Tsuyoshi Hayashi Harima Institute (Tetsutaro Iizuka, Dr.Sci.) Research organization (laboratories, research technology development office, etc.), Harima Research Promotion Division, Harima Safety Center Yokohama Institute (Tomoya Ogawa, Dr. Agr.) Genomics Sciences Center, Plant Science Center, SNP Research Center, Research Center for Allergy and Immunology, Yokohama Research Promotion Division, Safety Center Kobe Institute (Masatoshi Takeichi, Ph.D.) Center for Developmental Biology, Kobe Research Promotion Division, Kobe Safety Center 8-3 RIKEN Organization & Research Centers 18

  19. 8-4 Evaluation System of RIKEN RIKEN Overview RIKEN Advisory Council (RAC) = Evaluation of R&D Organization ・RAC: advisory body to the President on overall research activities and management in RIKEN Discovery Research Institute & Harima Institute Centers Evaluation of R&D Organizations Recommendation to Center Directors on research activities and management of Center. （Advisory Council） ・Internal and external evaluation by research themes (internal evaluation since 1965) Evaluation by external assessor of each research group as a unit Evaluation of R&D Themes ・External evaluation of research laboratories (research performance review since 1972) Evaluation conducted by Head of each research unit at the end of each fiscal year, when contracts are renewed. Evaluation of Researchers’ performance Internal evaluation committee to assess promotion for researchers (ex. research → senior researcher) The Evaluation System is essential to make appropriate judgments on R&D activities, hence to increase work efficiency and vitalize research activities to achieve higher goals.

  20. 8-5 Advisory Council System of RIKEN RIKEN Discovery Research Institute & Harima Institute Directors, DRI & HI AC RIKEN Advisory Council AC RIKEN Frontier Research System Director，FRS AC RIKEN Brain Science Institute Director，BSI AC RIKEN BioResource Center Director，BRC AC RIKEN Genomic Sciences Center Director，GSC AC RIKEN Plant Science Center Director，PSC Recommendation AC RIKEN SNP Research Center Director，SNP AC RIKEN Research Center for Allergy and Immunology Director，RCAI President The Board of Executive Directors AC RIKEN Center for Developmental Biology Director，CDB Review Report Advice Response

  21. 8-6 Distinctive Features of the Advisory Council System • Advisory Council (AC) Members are carefully, as well as internationally, selected from different academic fields to cover RIKEN’s broadly conducted research activities. • 　　　 → More than half of the members are foreign nationals. All five RAC meetings of the past were chaired by foreign nationals. • AC chairman of each Center to participate in the RAC meetings as a member. • → Appropriate for RAC which conducts umbrella evaluation of RIKEN. • Reports are produced only by AC members so that activities in Centers are directly and openly incorporated. • → Assurance of objectivity and fairness (however, RIKEN might present its arguments vis-à-vis those reports) • Responses to the RAC recommendations are compiled in a report immediately after the RAC meeting, which is sent to RAC Members. The final progress of actions taken to respond to the RAC recommendations is to be evaluated at the next RAC meeting. • → “Recommendation ⇒ Response ⇒ Reporting” makes a perfect flow to contribute to the RIKEN’s development.

  22. 8-7 Advisory Council Status (Past 5 Years)

  23. 8-8 What is the RAC? • The RIKEN Advisory Council conducts an overall review of RIKEN and makes recommendations and proposals to the President on managerial issues including management of the Board of Executive Directors. • Recommendations on policies to further vitalize research activities • Identifying research areas where resources should be increased and where further development is required • Recommendations on new areas of development, especially on how to embark on unexplored fields • Suggesting a policy to promote internationalization in RIKEN and international cooperation by RIKEN • Any other recommendations on other issues and measures for the advancement of RIKEN Meetings First meeting: 21-24 June 1993 Second meeting: 26-29 June 1995 Third meeting: 31 May - 5 June 1998 Fourth meeting: 4-7 June 2000 Fifth meeting: 6-9 June 2004 Sixth meeting: 6-9 June 2006 (tentative)

  24. 8-9 Terms of Reference to the 5th RAC Meeting - Noyori Initiative - • Visibility of RIKEN • Improve the image and public recognition of RIKEN • Researchers and administrative staff should make the public aware of the importance of science and technology • 2. Maintaining RIKENs Outstanding History of Achievement in Science and Technology • Sustain and build the RIKEN research spirit • Emphasize quality of research. Maintain excellence and prestige of the RIKEN brand • Strengthen action further to achieve intellectual properties and to make contributions to society and industry • 3. RIKEN that Motivates Researchers • Curiosity-driven concepts • Presenting challenges that are unique and high in risk level • Fostering promising human talent • 4. RIKEN that is Useful to the World • Close ties with industry and society • Science and technology that supports civilized society (beyond the boundaries of universities or industry) • 5. RIKEN that Contributes to Culture • RIKEN must raise the level of its own culture • Dissemination of knowledge to humanities and social science

  25. 8-10RAC Recommendations of the 5th RAC Meeting 1. Build a strong scientific vision for the future of RIKEN 1a) Revise the scientific governance structure 1b) Develop a long-term plan for the support of basic research activities at RIKEN 2.Strengthen the role of RIKEN President 2a) Set up an external advisory board reporting to the president 2b) Strengthen the position of the center and institute directors 3.Increase efforts to build strategic relationships 3a) Broaden the scope of internal strategic program 4. Develop programs to increase the quality of post-doctoral staff and graduate students working at RIKEN 4a) Create a pre-doctoral fellowship program to support graduate students working at RIKEN 4b) Provide support for all staffers seeking employment after their RIKEN contracts ends 5.Reassess RIKEN’s technology transfer regime 5a) Develop a strategic framework to support translational research 6. Increase the number of foreign scientists working at RIKEN 6a) Increase the number of Japanese women scientists in leading positions at RIKEN 7.Implement best practices in management and administrative services 7a) Develop a coherent long-term personnel managementstrategy 7b) Develop best practices for the management of large facilities and resource collections

  26. 8-11 Issues under Evaluation ・Pressure from Evaluation Evaluation imposes pressure on researchers, especially in research organizations consisting of Fixed-term Contract Researchers. This seems to be increasing tensions among researchers beyond what is necessary. 　　→　Discourage researchers to take on challenges ・Combine information overlapped among various evaluation activities The common information such as research performance is repeatedly gathered by different persons for various evaluation activities as the information is required at a different timing and by different evaluation organizations, i.e. IAI (Independent Administrative Institution) Evaluation and RAC Meeting. Having that in mind, it is advisable to establish an efficient data-management system for combining and/or sharing such common information to avoid duplicate work and reduce workload of both researchers and back-office staff.

  27. 9. Evaluation System of NIMS9-1 Profile of NIMS Basis for Establishment NIMS Law (Independent Administrative Institution) Date of Establishment April 1, 2001 Mission 1. Conduct “Basic” and “Generic and Infrastructural” research 2. Promote dissemination of research outcomes and their utilization 3. Open advanced facilities and equipments to outside researchers 4. Educate materials scientists and oegineers 2005 budget $173.5 million (of which $146.5 million is funded from the government subsidy and $23.2 million is income for entrusted research (ex. Competitive fund)) Employees as of end of 2004 1,538 full-time employees (of which 692 are permanent staff) President KISHI Teruo, Dr. Eng. Focused R&D areas under Five-Year Program at MINS (2001-2006) Nanomaterials Safe Materials Environment and Energy Materials Improvement of Research and Intellectual Infrastructure

  28. 9-2 Organization of NIMS Advanced Materials Laboratory Nano-materials Laboratory Materials Engineering Laboratory President Biomaterials Center Superconducting Materials Center Computational Materials Science Center Nanotechnology Researchers Network Center of Japan Steel Research Center Ecomaterials Center International Center for Young Scientists High Magnetic Field Center Materials Information Technology Station Cooperative Graduate School NIMS-Course, Univ. of Tsukuba Material Analysis Station High Voltage Electron Microscopy Station

  29. 9-3 Schematic Image of Evaluation System in NIMS 3. MIAC Evaluation Committee for Policy and IAI 1. Evaluation of NIMS ・ by MEXT IAI Committee (Annual Evaluation of the Mid-term Program Achievement) Secretariat: MEXT 4. R & D Evaluation Submit self-rating sheets ・based on “R & D Evaluation Procedure” 2. Making Self-rating Sheets by NIMS Secretariat: ISO and REO i) R&D Projects by External Funds ・ by Funding Agency Evaluation Results are Reflected on Management and R & D Activities through Reallocation of Budget by the Executive Meeting ii) Major Research Projects ・by External Committee(advance, mid-term, ex post facto and follow-up) iii) Exploratory Researches ・by Director General of Each Unit (advance and ex post facto) 5. Researchers Personal Evaluation 6. NIMS and Research Unit Evaluation Secretariat: REO ・based on “Researchers Personal Evaluation Procedure” ・by the NIMS Advisory Board iv) R&D projects by NIMS Competitive Funds Secretariat: ISO and REO Secretariat: ISO ・by Internal Committee (advance and ex post facto) Use of Evaluation Results Secretariat: ISO Reflection of Evaluation Results MIAC: Ministry of Internal Affairs and Communications R & D: Research and Development MEXT: Ministry of Education, Culture, Sports, Science and Technology ISO: Integrated Strategy Office IAI: Independent Administrative Institution REO: Research Evaluation Office

  30. 9-4 Headings of NIMS evaluation as an independent administrative institute (1) • Management of research organization • Improving the service to the people and other management • I. 1. Basic research and development • 1. 1 Major research projects • I. 1. 1. 1 Nanomaterials • I. 1. 1. 2 Environment and energy materials • I. 1. 1. 3 Safe materials • I. 1. 2 Improvement of research and intellectual infrastructure • I. 1. 2 1) Improvement of research infrastructure • I. 1. 2 2) Improvement of intellectual infrastructure • I. 1. 3 Exploratory researches • I. 1. 4 Proposal to external funds and acceptance of entrusted researches

  31. I. 2. Dissemination of research outcomes and their utilization I. 2. 1 Disseminationof research outcomes and promotion of public relations I. 2. 1 ① Research products I. 2. 1 ② Public relations I. 2. 1 ③ Publishing materials data bases and data sheets I. 2. 2 Promotion of technology transfer Number of patent applications Number of licensed patents I. 3 Making facilities and equipments open to non-NIMS researchers

  32. I. 4 Training and education of researchers and technicians I. 4. 1 Reception of trainees I. 4. 2 Participation and dispatch of lecturers to scientific societies and academic meetings Number of participants to the scientific societies and academic meetings Number of lecturers dispatched to scientific societies and academic meetings I. 5 Others I. 5. 1 Improvement of survey and coordinate functions I. 5. 2 Research exchanges I. 5. 2 1) Implementation of joint research and promotion of cooperation I. 5. 2 2) Reception of researchers from other institutes I. 5. 2 3) Dispatch of NIMS researchers to other institutes I. 5. 3 Cooperation in accident-cause-finding

  33. II. Steps to achieve goals for efficient management II. 1. Organizational and management systems II. 1. 1 Fundamental policy for research organization II. 1. 1 ① Construction of research system II. 1. 1 ② Construction of research support system II. 1. 1 ③ Construction of technology transfer system II. 1. 2 Fundamental policy for management II. 1. 2 1) Expansion of project leaders discretion II. 1. 2 2) Rational arrangement of human resources II. 1. 2 3) Efficiency of management II. 1. 2 3) ① Simplicity, rapidity and efficiency of administrative procedures II. 1. 2 3) ② Outsourcing II. 1. 2 3) ③ Efficient management of operating subsidy

  34. III. Budget, income and expenditure and fund plan IV. Amount of short-term loan payable V. Plan for disposal or mortgage of important assets if planned VI. Use of surplus fund VII. Others concerning management specified in MEXT ordinance VII. 1. Planning for facilities and equipments VII. 2. Planning for personnel affairs

  35. 9-5 Outline of Evaluation of Major Research Projects (The NIMS rule is based on the government outline and MEXT guidelines for evaluation.) Times of evaluation： Advanced/ mid-term (for 5-years projects)/ ex post facto/ follow-up (for selected projects) (25 Major research projects are evaluated.) Members of evaluation committee: (1) Candidates of about 10 specialists outside NIMS are proposed. (2) Persons concerned are excluded. (3) 4 or 5 are to be selected. Preparation: Materials for evaluation and evaluation sheets are sent beforehand to committee members.

  36. Evaluation committee: Project leaders explain the aim, objective, plan, results and so on. Committee members and the project leaders have discussion. Evaluation results: Evaluation sheets compiled by the chairperson are submitted to the project leaders. If the leader is against the results, discussion will be held between committee members and the project leaders through the secretariat. Reflection of results: The president of NIMS confirms the final evaluation results. The results are reflected on the budget and to improve the project. Official announcement: The final evaluation results are disclosed to public via internet.

  37. 9-6 Performance-based Personnel System Objective Maximize the research achievements under a competitive environment to accomplish the institute’s goals Framework Develop achievement-oriented pay and results-driven achievement systems Reward high-performing researchers Achievements to evaluate 3P (Papers, Patents, Products) Scored automatically by on-line system Paper = 3 + Impact factor x 2 Activities contributed to NIMS’ missions except 3P （management, academic activities, collaborations, research assistance, etc.） Rated by supervisor

  38. 9-7 Performance-based Pay System • Both monthly pay and bonus reflect Individual achievements • Performance-based pay: determined by individual achievements over last several years • Director-general: 18 ~ 40 % of basic pay • Director: 14 ~ 25 % • Senior researcher: 12 ~ 20 % • Performance-based bonus: determined by individual achievement of the year before • Director: 4.25 months + a (average: 0.4 months) • a is proportional to individual achievement point

More Related