A Study of the Model and Methodology for Institute Evaluation in CAS

1. A Study of the Model and Methodology for Institute Evaluation in CAS Xiaoxuan Li，Bing Shi，Jianzhong Zhou Institute of Policy and Management，CAS Management Innovation and Evaluation Research Center, CAS

2. Contents • Brief introduction of CAS and Evaluation Practice in CAS Institutes • Three-hierarchy Evaluation Model • Conclusions

3. 1 Brief Introduction of CAS and Evaluation Practice in CAS Institutes

4. The Chinese Academy of Sciences (CAS) was founded in Beijing on November 1, 1949. Consisting of the Academic Divisions and various subordinate institutions, it is the lead national academic institution in natural sciences, a major advisory body to the government on science and technology related issues, and a national comprehensive research and development center in natural sciences and high technology areas in China. The general objectives of CAS are to develop into a base for scientific research, for training high caliber scientific talent and for incubating high-tech industries in China; to become a national scientific think tank and to evolve into a national research institution that boasts “first-class achievements, first-class efficiency, first-class management and first-class talent.” Brief introduction of CAS

5. Overview of CAS CAS Academic Divisions CAS Headquarters Institutions Directly under CAS Main Infrastructure • Division of Mathematics and Physics • Division of Chemistry • Division of Life Sciences and Medicine • Division of Earth Sciences • Division of Information Technical Sciences • Division of Technological Sciences － 17 Large-scale Scientific Research Facilities － 7 National Labs － 5 Field Stations Networks － 36 National Engineering Centers － 273 Technology Transfer Centers － 317 Journals － 46 National Associations and Societies －94 Research Institutes －2 Universities and Schools －2 Supporting Units －3 Botanical Gardens －12 Branches －2 Press and Publication Companies －1 Assets Management Company －22 Holding Enterprises Committee for Consultation and Review Committee on Scientific Ethics Committee for Science Popularization and Publication Distributed in 22 provinces and cities over China. Members 709 Foreign Members53

6. In 1998, with the approval of the Chinese government, the CAS launched the Knowledge Innovation Program (KIP), and has since turned over a brand new page in its history of development. In the process of conducting the KIP, in recognition of the national strategic requirements and the world trend in science and technology, the CAS has made the most profound and extensive adjustments in its disciplinary deployment and organizational structure since its founding in 1949. Scientific restructuring and management innovation has led to the preliminary formation of a structure and mechanism that are both geared to a socialist market economy and aligned with international standards.

7. Changes of Concepts in Institutes Evaluation in CAS 15Years/3Period • From 1993 to 1998, the institutes evaluation in CAS was focused on research outputs ,it is called output evaluation. • From 1998 to 2004, the CAS launched the Knowledge Innovation Program (KIP), the institutes evaluation in CAS was focused on achievements and level of accomplishment, it is called performance evaluation. • From 2005 to now, the institutes evaluation in CAS was focused on S&T creativity and concerned performance in integrity, it is called quality-orientated comprehensive evaluation.

8. 2 Three-hierarchy Evaluation Model

9. Comprehensive Complementary Reform Pilot Project In 2007, seven pilot institutes have been selected to investigate modern R&D institutions system Four thematic issues haven been launched in the following aspects respectively: Resource Allocation, Human Resources, Organizational model of R&D, Evaluation. Basic Tasks of Evaluation Issue: To study evaluation models for different types of institutes. Background and Tasks

10. Categories of 7 Pilot Institutes Institute of High Energy Physics Institute of Computing Technology • Large scale scientific facilities • High-tech Research Cold and Arid Regions Environmental & Engineering Research Institute Dalian Institute of Chemical Physics • Comprehensive institute • Resources, Environment and institute in western China • Biological research Shanghai Institute of Technical Physics Institute of Microbiology • Engineering and big teams Institute of Physics • Basic research

11. Three-hierarchy Evaluation Model At the top hierarchy is qualitative evaluation by experts review, and it is also a hierarchy of individual evaluation. Experts review include two stages: diagnosing evaluation for individual institutes and comparing evaluation among the institutes of the same category. Peer review At the second hierarchy are the key indicators and benchmarks, which reflects the characters of certain category of institute. It can be used in both diagnosing evaluation for individual institute and comparison between institutes of the same category. Key indicators Quantitative Qualitative At the bottom hierarchy are indicators that can reflect common characters of the institutes, including yearly quantitative monitoring and yearly communication review Yearly quantitative and qualitative monitoring

12. Key indicators and benchmark The selection of key indicators: Based on the orientation and characteristics of institutes, we selected no more than 6 key indicators for each pilot institute. Totally, there are 3 common key indicators and 8 specific key indicators. There are four methods to set the benchmark in our research : 1) the more the better, for example the more awards, the better; 2)different standards will be given based on international comparison; 3)bechmark will be set according to the development level of institutes; 4)judged by experts experience.

13. key indicators of 7 pilot institutes 共性指标： 不同研究所的共性因素

14. The methods to set common key indicators of benchmark

15. Specific key indicators are determined by separately by the 7 pilot institutes. The study is expected to be accomplished by the end of this year. The methods to set the specific indicators of benchmarks

16. Conclusions ——Through the study of 7 pilot institutes, key indicators and benchmarks regarding different types of institutes were proposed, including key quantitative indicators and benchmarks as well as key qualitative indicators and anchoring method. Thereby, three hierarchy evaluation model has been set up. ——the new model puts more emphases on the character of evaluation by category and individual features of institutes. —— In summary, this model has developed Quality-orientated Comprehensive Evaluation system in methodology. 3 Conclusions

17. Thanks Contact information: www.casipm.ac.cn jzzhou@casipm.ac.cn