Performance Indicators for Decision Making in Academia

1. Towards the use of performance indicators in decision making CCDS, St John’s, Newfoundland John RG Challis FRSC Vice President Research and Associate Provost Professor of Physiology, Obstetrics and Medicine University of Toronto

2. Performance Indicators • Increased funding to academia • Increased requirement for accountability • Public return on investment • Efficiency and effectiveness of use • Quality

3. Quality and Impact Research Quality Knowledge generation, manifestation, acceptance, expertise Research Impact Knowledge diffusion, production, relationships, engagement Research capability Knowledge infrastructure, culture, training janet halliwell, 2006 janet halliwell, 2006

4. Performance Indicators • Internal planning, allocation of resources and outcome evaluation • Governance assurance • National/international comparisons • Government/agency accountability • Advocacy

5. HQP Publications/Citations Honours Social Impact Commercialization Funding Great Minds University ResearchInputs and Outputs

6. HQP Publications/Citations Honours Social Impact Commercialization Funding Great Minds University ResearchInput: FUNDING HQP Publications/Citations Honours Social Impact Commercialization FUNDING Great Minds

7. Research Funding Trend University of Toronto & AffiliatesResearch Funds Awarded Excludes: funding from the Research Performance Fund and the federal Indirect Costs program.

8. University of Toronto and AffiliatesResearch Funds Awarded, 2003-04 *Excludes all known indirect costs (for funding administered at affiliated hospitals, this is usually not known)

9. Research Expenditures, 2002-03In USD ($M)

10. Five Ways to Look at Funding • Current Dollars • Constant Dollars • Share (among Canadian colleges & universities) • (Current) Dollars per Eligible Faculty • Research Yield (share of funding received by an institution relative to its share of eligible faculty in the respective disciplines.)

11. Federal Research Council Payments Until 2004-05 (Current $)University of Toronto and Affiliates Percentages on right are for 2004-05

12. Federal Research Council Payments Until 2004-05(Constant 2005 $)University of Toronto and Affiliates Constant 2005 dollars using the consumer price index

13. NSERC Funding to U of TConstant and Current Dollars Source: Current dollar data: NSERC Facts & Figures; constant 2005 dollars calculated using the consumer price index (CPI).

14. Distribution of Funds Awarded by AgeUniversity of Toronto & Affiliates, 2003-04

15. Distribution of Funds Awarded by AgeUniversity of Toronto & Affiliates, 2003-04

16. Distribution of Funds Awarded by AgeUniversity of Toronto & Affiliates, 2003-04

17. Faculty of MedicineResearch Funds Awarded Source: Office of the Vice-President, Research and Associate Provost

18. Funding Profile Faculty A

19. Research Funding from the Tri-Councils to UniversitiesIn Constant 2003 Dollars Indexed to 1977 to 1979 Total Data source: CAUBO (Income from sponsored research)

20. Research Funding from Provincial Sources to UniversitiesIn Constant 2003 Dollars Indexed to 1977 to 1979 Total Data source: CAUBO (Income from sponsored research)

21. Ontario’s Share of Research Funding Compared to All Canadian Colleges and Universities Sources: CAUBO and Councils (Councils exclude NCE & CRC).

22. NSERC Funding per Eligible Faculty for G10 Not shown: Eight G10 institutions with NSERC funding per eligible faculty lower than $80,000 in 2004-05: Alberta, British Columbia, McGill, Laval, Montréal, McMaster, Waterloo, Western.

23. Research Yield = Institutional Research Funding for Faculty National Research Funding for Faculty Institutional Faculty Count National Faculty Count Share of funding received by an institution relative to its share of eligible faculty in the respective disciplines. For NSERC Research Yield, data on funding for faculty come from NSERC’s Facts & Figures and exclude training programs, Canada Research Chairs, Networks of Centres of Excellence and Indirect Costs. Eligible faculty counts are based on data from Statistics Canada.

24. NSERC Research Yield for G10 Not shown: Eight G10 institutions with yields lower than 1.6 in 2004-05: Alberta, British Columbia, McGill, Laval, Montréal, McMaster, Waterloo, Western.

25. Tri-council Funding • Gold Standard, excellence and credibility • Determines numbers of Canada Research Chairs • Determines amounts of indirect costs – full cost recovery of research

26. Indirect Costs Rate Vs Direct Funding 2006-07 Includes additional $40M

27. Indirect Costs Rate Comparison (1) Harvard affiliated hospitals (2) AAUDE: Association of American Universities Data Exchange (3) CIDA rate applies to payroll costs (4) U of T federal Indirect Costs Program, 2006-07, after $40M increase

28. Research Indirect Costs ShortfallPer $100 Direct FundingU of T Excluding Affiliates, 2003-04

29. HQP HQP Publications/Citations Publications/Citations Honours Honours Social Impact Social Impact Commercialization Commercialization Great Minds Funding Funding Great Minds University ResearchInput: GREAT MINDS

30. Research Clusters – All In y/x, y = current number of Chairholders; x = planned number of Chairs Cluster has planned Chairs but no ChairholderCluster has no planned Chair and no current Chairholder

31. Canada Research Chairs’ OriginU of T and Affiliates

32. Canada Research Chairs’ GenderU of T and Affiliates

33. Personnel Awards & Designations U of T and Affiliates, 2003-04 “Other” includes corporations, institutions and international organizations.

34. Personnel Awards & Designations U of T and Affiliates, 2003-04 “Other” includes corporations, institutions and international organizations.

35. Implications of CIHR Personnel Awards Cuts to U of T

36. HQP Publications/Citations Honours Social Impact Commercialization Funding Great Minds University ResearchOutput: PUBLICATIONS

37. All Fields – Number of PublicationsTop 10 Public AAU and All G13 Institutions, 2001-2005 Source: U.S. and Canadian University Indicators, Deluxe Edition 2005, Thomson Scientific.

38. All Fields – Number of CitationsTop 10 Public AAU and All G13 Institutions, 2001-2005 Source: U.S. and Canadian University Indicators, Deluxe Edition 2005, Thomson Scientific.

39. All Science Fields – Number of PublicationsTop 10 Public AAU and All G13 Institutions, 2001-2005 Source: U.S. and Canadian University Indicators, Standard Edition 2005, Thomson Scientific.

40. All Science Fields – Number of CitationsTop 10 Public AAU and All G13 Institutions, 2001-2005 Source: U.S. and Canadian University Indicators, Standard Edition 2005, Thomson Scientific.

41. Most Commonly Used Tools for Bibliometrics All by Thomson Scientific • University Science Indicators (UI) • Essential Science Indicators (ESI) • Citation Indexes • Web of Science

42. Comparison of UI and ESIPublication Count 1995 to 2004 All Fields U of T’s Rank among North American public and private universities UI ESI 2nd5th UI: University Indicators: highly aggregated publication and citation counts by university. Universities can submit their list of affiliates and Thomson will count affiliates’ numbers with parent university. ESI: Essential Science Indicators: web-based query tool where institution data come from authors’ addresses. Not possible to correct for missing affiliations in addresses.

43. Publications of Authors from Hospitals Affiliated w/U of TAnalysis of Author’s Address in 2003-04 * Hospital only address for one author, U of T address for another. Source: Web of Science, for publications between January 2003 and April 2004.

44. Examples of Product Use • UI: VP RAP Annual Report (#2 ) • UI: Science Watch Canadian University Comparison Fall 2005 (various ranks in 21 fields) • ESI: Times Higher Education Supplement Rankings (#29)

45. ISI-Highly Cited Researchers in Canadian Universities ISI-Highly cited researchers represent the 250 most highly cited researchers in each of 21 broad subject areas. Data above dated May 2006.