1 / 23

Peer Review in CAS Institute Evaluation

Peer Review in CAS Institute Evaluation. Zhou Jianzhong , Zhou Changhai , Li Xiaoxuan Institute of Policy and Management, Chinese Academy of Sciences Assessment Office, Bureau of Planning Strategy, Chinese Academy of Sciences 2012.10.24-27 Minneapolis. Outlines.

jacquelinen
Download Presentation

Peer Review in CAS Institute Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Peer Review in CAS Institute Evaluation Zhou Jianzhong, Zhou Changhai, Li Xiaoxuan Institute of Policy and Management, Chinese Academy of Sciences Assessment Office, Bureau of Planning Strategy, Chinese Academy of Sciences 2012.10.24-27 Minneapolis

  2. Outlines • A brief introduction of CAS • The practices of peer review in the evaluation of institute of CAS • A New model of Peer review in CAS institutes: “ONE-THREE-FIVE” Expert Diagnosis Assessment • Discussion and conclusion

  3. History & Position 1949 2010 CAS was Founded on Nov. 1, 1949 ◆ Highest academic institution in natural sciences in China ◆ A national comprehensive research and development center in natural sciences and high technology areas in China ◆ A major advisory body to the government on science and technology

  4. Overview of CAS CAS 2011 Staff：60700 Graduate Students：49000 Academic Divisions CAS Headquarters Institutions Directly under CAS Main Infrastructure • Division of Mathematics and Physics • Division of Chemistry • Division of Life Sciences and Medicine • Division of Earth Sciences • Division of Information Technical Sciences • Division of Technological Sciences － 17 Large-scale Scientific Research Facilities － 7 National Labs － 5 Field Stations Networks － 36 National Engineering Centers － 273 Knowledge Transformation Centers － 317 Journals － 46 National Associations and Societies －100 Research Institutes －2 Universities and Schools －2 Supporting Units －3 Botanical Gardens －12 Branches －2 Press and Publication Companies －1 Assets Management Company －22 Holding Enterprises Committee for Consultation and Review Committee on Scientific Ethics Committee for Science Popularization and Publication Members 709 Foreign Members53

  5. Landscape of CAS affiliates（2010） Jilin（3） Xinjiang（2） Beijing （40） Liaoning（4） Tianjin（1） Gansu（3） Shanxi （1） Shandong（3） Qinghai（2） 12branches Jiangsu（6个） • Beijing Branch • Shenyang Branch • Changchun Branch • Shanghai Branch • Nanjing Branch • Wuhan Branch • Guangzhou Branch • Chengdu Branch • Kunming Branch • Xi‘an Branch • Lanzhou Branch • Xinjiang Branch Shanxi （3） Shanghai（11） Sichuan （3） Hubei （5） Anhui （1） Zhejiang（1） Guizhou （1） Hunan （1） Fujian （2） Yunnan（2） • CAS affiliates could be found in 27 cities in 22 provinces, autonomous regions and municipalities directly under the central government Guangdong （5）

  6. Layout of the main subject areas and research fields • Basic Research • Life Sciences and Biotechnology • Resources and the Environment • High Technology Four Aspects Coverage of the natural science and high technology of the main research fields • Material Science and Technology • Life Science and Technology • Space Science and Technology • Earth Science and Technology • Information Science and Technology • Math and Big Science Facilities

  7. The development of institute evaluation in CAS 15Years/3Period Changes of Concepts in Evaluation 2005～2010 1999～2004 1993～1998 Output Evaluation • Focus on research outputs Performance Evaluation • Focus on achievements • Focus on level of completion Comprehensive Quality Evaluation • Focus on S&T creativity • Care performance in integrity Encouraging competition, enhance innovation capability and efficiency by performance evaluation and process control. Policy tool Analyzing tool Managing tool Incarnates guidance & inspiring Compare & construe states of development Control & management of process

  8. Changes of Evaluation Concepts and Methods of Institutes in CAS Give priority to quantitative evaluation Qualitative evaluation Ranking of institutes Group by sort Isolated quantitative evaluation and peer-review Integration of quantitative quality assessment and peer-review Simple external evaluation Combination of self-evaluation & focus group discussion Comprehensive evaluation of Performance, management process, future development Performance evaluation only Evaluate every five year Annually

  9. The practice of peer review in the evaluation of institute of CAS

  10. Experts selection criteria and mechanism criteria : • Have considerable popularity in certain discipline • Have generally experience in research management • Currently in the first-line of research mechanism : • The institute recommends candidates of about two times the number of review experts wanted • The profession office of CAS authorities reviewed the list of experts • Inviting experts in the name of the president of CAS to participate in the review

  11. Organization and procedural arrangements Communication assessment: • The Evaluation Research Center(ERC) contacts the assessment experts that have been determined, sends evaluation materials and questionnaires • Assessment experts give assessment feedback within the specified period • The assessment center organizes the review results to form the assessment report On-Site assessment: • CAS leaders introduce the basic situation of CAS and the background and requirements of assessment to international assessment experts • The experts group hear reports of the institute director and of a number of academic leaders • Visit labs and discuss with scientists and graduate students • The experts group carries out a closed meeting to communicate with the leading body

  12. The nature of scientific research activities of different research need different experts The peer review system is suit to evaluate the basic research and the experts is easy to select from international. However, the peer review isn’t suit to evaluate the applied research and high-tech research, which meets national strategic needs and national economic development. Perhaps some research work is low levels by experts’ opinion, but for China's economic and social development is of great significance, such research work should not invite international experts. Issues

  13. How to select the experts and provide the criteria of evaluation The selection of experts, as well as the evaluation criteria of scientific and technical work have been always plagued us. The International criteria accepted peer-reviewed in the scientific community is mostly innovative in the science and technology. This is the base for compare the different research work. However, The general objectives of CAS are to develop into a base for scientific research, for training high caliber scientific talent and for incubating high-tech industries in China; to become a national scientific think tank and to evolve into a national research institution that boasts “first-class achievements, first-class efficiency, first-class management and first-class talent.” a lot of work of CAS are to meet national needs. Therefore, evaluation criteria can’t only be just the scientific research criteria, the evaluation experts can not only be scientists.

  14. How to make use of the results of evaluation The results of experts evaluation are not comparable because different experts use the different evaluation criteria. Some experts will give the lower evaluation that refer to the highest international research level criteria . Some expert will give the higher evaluation with reference to the relevant domestic research level criteria. The discrepancy of expert evaluation results was less and was difficult to distinguish grade. Because the experts was recommended by institute and have the better relations with the institute. So these experts are easy to give a good evaluation result. So the management is difficultly to makes use of the result of the peer review.

  15. A New model of Peer review in CAS institutes: “ONE-THREE-FIVE” Expert Diagnosis Assessment • In 2010, the innovation capacity of CAS institutes had been greatly improved, the level of the staff was advanced and the institute entered a new stage and class in the CAS. The 105th executive meeting of the State Council fully affirmed the achievements made by KIP and adopted the “Innovation 2020” of CAS. • Innovation 2020 aims at solving major scientific and technological problems in basic, strategic and prospective research fields that concerning the overall and long-term development of the country. • In order to fulfill the goal of “Innovation 2020”,CAS put forward new strategies in the new developing era. That is “one-Three-Five” strategy： One Positioning, Three Major Breakthroughs, Five Cultivating Directions.

  16. The Twelfth Five-Year Development Plan During the Twelfth Five-Year (2011~2015), required as the strategic planning called “One-Three-Five”, each institute of CAS will promotes its develop plan，identify its position, and define its research priorities with potential breakthroughs and its prospective layout. One • One Positioning indicates each CAS institute should specify its major research areas, unique features, core competitiveness, and anticipated position in international circles and should avoid homogenization with other CAS research institutes. Three • Three Major Breakthroughs indicate major basic, strategic and prospective S&T innovative achievements to be made in the next five to ten years; generally each institute shall raise no more than 3 breakthroughs. Five • Five Potential Directions indicate research priorities with unique features, a future competitive advantage and potential breakthroughs; generally each institute shall set no more than 5 priorities.

  17. Clarifying the value guidance：6 types of major innovation contribution and innovation in talents and mechanism 1 During the 12th five-year plan，CAS has developed a major R&D outcome-oriented evaluation system • Key Scientific Issues • Open new directions • Key Core Technology • Systematic Solution • Significant Social and Economic Benefits • Great influence on decision making Designing reasonable assessment process 2 • Judging performance • Diagnosing and monitoring key element which is the benefit of performance

  18. The major research outcome-oriented • evaluation system of CAS Sections Aim structure During2015 Checking target completion of key breakthrough • Based on the assignment book • To judge the target completion • Emphasizing key breakthrough 2 sections During 2013-2015 “1-3-5” Expert diagnosis assessment • Diagnosing key areas and factors • Guiding the development of the institutes • Driving the implementation of “1,3,5” yearly Monitoring key indicators of institutes yearly • Holding the changing and development of core competitiveness • Serving expert diagnosis assessment 1 base Supporting platform Theory tank、tank of methods and tools、expert tank、database，network research organization……

  19. Objective and role of “One-Three-Five” expert diagnosis assessment Inviting high level experts domestic and overseas • Diagnosing the positioning, advantages and disadvantages of the institutes • Diagnosing the research quality and technical value of the key fields, research significance and impact, talents, resources and technology platform etc. • Diagnosing the positioning, core competitiveness of the institutes and their position among relative institutes around the world Through the diagnosis assessment • Help the institutions to improve management，clarify core advantage、avoid homogenization • Lay a foundation for the future major innovation contribution

  20. Discussion and conclusion • As a national research institution, each institute in CAS has its own mission and position. What kinds of expert can evaluate not only research quality but also the research work that meets the national strategic needs? So it needs the peer has different type. The peer of the institute evaluation is not only to judge the research quality based on international level, but also from the perspective of users invited experts to evaluate their capacity of meeting the national need. • Some institutes in CAS are the comprehensive research institutes which include basic research and applied research. That leads to institutions in CAS covering many disciplines and fields. So, how to ensure the peers can evaluate every discipline and field in the institute while controlling the expert group proper scale? • There are so many institutes in CAS and in different development levels. Some institutes in high level can be evaluated and measured by international benchmark. However, when some institutes are still in a relatively low level, it is difficult for making a judgment.

  21. Summary of the “1-3-5”Expert diagnosis assessment • The object and content of review are more focused • In the institute peer review of the major research outcome-oriented evaluation system of CAS, the diagnostic assessment is carried out for each institute every five years by domestic and foreign high-level peer experts and user experts mainly from the international perspective and national needs. The experts do not assess the entire institute or diagnose the entire research staff, but focus on the advantages and core competencies of the institute and judge the advancement of the work of major fields. • The assessment results focus on diagnosis but are not linked with resource allocation • In CAS’s new peer review, the assessment results are not linked with the allocation of resources but focus on diagnosis and help the institute to identify problems, grasp the direction, make a rational layout, improve management and enhance the efficiency of achieving significant achievements output. At the same time, the results are used to promote the exchanges between research staff and high-level experts in the same field and form the academic atmosphere of pursuing excellence and daring to innovate. Therefore, the institute has small pressure in the process of evaluation without too much burden and can organize the evaluation easily. This is a large difference from the previous peer review.

  22. Challenges • How to determine the criteria of Major Outcome? • How to select proper experts? • Can expert judge the Major Breakthroughs? • ……

  23. Thanks！ Contact: jzzhou@casipm.ac.cn

More Related