Multi-Dimensional Factors in Academic Research Evaluation

1. Multi-Dimensional Factors in Academic Research Evaluation Xiaodong Zhang Ohio State University

2. Impact of Strong Research to Universities • A foundation of improving undergraduate and graduate programs • Timely updating contents in classrooms • Creating research environment and opportunities to students • Bringing competitiveness and recognition • New knowledge, discoveries, and innovations. • Continue to attract strong students, excellent faculty, and research grants.

3. How is Research Measured? • Research Production is Typically Measured by • Publications in refereed conferences and journals • Software and prototypes • Invited lectures • Patents • Research Awards and honors • Input of Research • Research proposal writing, and research project planing • Research grants • Recruiting and forming teams.

4. How is Research Impact Measured? • Research results are useful • Citations of the publications • Widely used software and prototypes • Technology transfer from original research results • Useful patents • Influence in the research community • Leading new directions in the field • The quality of research is essential • Number of publications should not be a major metric • Our research should be Impact-driven

5. Where to Publish? • Why publish in top venues? • Each field has its own flagship and leading conferences/journals defined by reputations • Most researchers only read papers there. • Conferences versus Journals • In certain fields, such as systems, architecture, networking, and databases, top conference papers are prestigious, highly visible, with 4-5 or more reviews in depth, and low acceptance rate. • SCI and EI should NOT be used as a guidance • An SCI/EI entry does not necessarily reflect its quality in the field • The quality should be judged by the peers in the field. • The impact factor sometimes is field-size dependent.

6. Establishing Identity and Reputation • Conducting research on focused topics in depth • Making names behind some important innovations • US NSF Career Program for Junior Faculty • A 5 year research/education plan, aiming at conducting research to solve a small set significant problems. • A strong department should have several identities • Paying attention to building on existing strength. • Rewarding to strong research identity/reputation

7. What is an ``Acceptable” Research Production? • How to quantify research productions? • Number vs. quality of publications • Research grants/expenditure (mission vs. basic research) • Supervising Ph.D. students: productivity and quality of placements. • Impact of research after many years. • Conference/journal organizations • Not part of the research, but professional service. • Each department has its formula • Resources dependent (teaching load, et. al.)

8. Effects of University Rankings • Rankings by society have to be a consideration • Rankings significantly affect student recruiting • University rankings influence parents • Faculty recruiting is also affected. • However, these rankings are a reference not an absolute indicator.

9. Major Sources of University Ranking in US • US News & World Reports (University ranking) • Published every August. • Based on 6 criteria to quantify comprehensive ability. • Quality of undergraduate students is an important consideration. • US News & World Reports (Graduate school ranking) • Published every March. • Different school rankings are based on different disciplines. • Quality of graduate programs is the major factor. • National Science Foundation (funding statistics) • Research expenditure of last year published every August • Ranking the research scale of universities. • National Research Council (Department ranking) • Published every 10 to 15 years • Ranking all the Ph.D. granted departments in all majors

10. US News & World Reports’ 6 Criteria • Undergraduate student selectivity • SAT scores, high school ranking, and other activities. • Acceptance rate. • Faculty academic reputation • Research activities and accomplishments • Faculty-student ratio • Reflecting the efforts and attention to students, e.g. class size. • Student retention and graduation rate • Seriousness of the students and management of curriculum. • Financial resources • University revenue and endowed funds. • Alumni donation rate • Reflecting graduates’ proud, passion, and care to the university.

11. Major Themes of Graduate Program Ranking • Scholarships and impacts • Quality of faculty publications and their influence to the field • Citations rates and technology transfers • Ph.D. student production and placement • How many per year, and where they go after graduations. • Ph.D. alumni achievements • The group distinguishing themselves in the fields. • Faculty resources • Award winners, members of the academies, and others. • Research scale • Measured by the amount of expenditure by external grants

12. Foundations of Academic Excellence • Excellent undergraduate programs • Highly selected students with comprehensive talents. • Providing highly quality education and research environment. • All top U.S. universities have top undergraduate programs. • Excellent research and education infrastructure • Students are able to effectively learn and research. • Excellent faculty • World-class professors who are well established in their fields. • High standards in research and education • Rigorous training to students and doing impact driven research. • Ambitious students who will play leadership anywhere • Enter: grow the wisdom; Depart: serve the better country and the kind. • Should not be too narrowly focused.

13. Mr. Hu’s 4 Conditions of an ``Established University” • In 1947, Hu Shi urged the government to invest 5 existing universities to make them be ``established and independent” in the world within 10 years. • 1. Providing first-class education environment. • 2. Being able to retain young researchers. • 3. Being able to solve country’s problems in science, industries, health, and national defense. • 4. Being able to collaborate with foreign scientists to solve open and hard problems.