The Multi-Dimensional Research Assessment Matrix

1. The Multi-Dimensional Research Assessment Matrix Henk F. Moed Elsevier, Amsterdam, the Netherlands Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012

2. Short CV Henk F. Moed

3. Contents

4. (o)Bibliometrics makes sense

5. Which country has these main collaborators? USA

6. Which country has these main collaborators? Brazil

7. Which country has these main collaborators? Malaysia

8. Which country has these main collaborators? Romania

9. Which country has these main collaborators? SouthAfrica

10. Contents

11. (i) Bibliometric research assessment is potentially a proper tool to consolidate academic freedom

12. (ii) Bibliometric tools help establishing a longer term perspective in research funding

13. (iii) One must be cautious using “societal benefit” as an assessment criterion of basic research: it can not be measured in a politically neutral way

14. (iv) Citations measurescientific-scholarly impact rather than quality or validity

15. (v) Citation counts in social sciences and humanities may be influenced by politicalideologies

16. Citation impact and ideology Fall of the Berlin wall in Nov. 1989

17. (vi)The future of research assessment lies in the intelligent combination of metrics and peer review

18. (vii)Case study on funding policies of a National Research Council reveals biases in peer review

19. Affinity Applicants – Evaluation Committee 0 Applicants are/were not member of any Committee • Co-applicant is/was member of a Committee, but notof the one evaluating • Firstapplicant is/was member of a Committee, but not of the one evaluating • Co-applicant is member of the Committee(s) evaluating the proposal • Firstapplicant is member of the Committee(s) evaluating the proposal

20. For 15 % of SUBMITTED applications an applicant is a member of the evaluating Committee (Affinity=3, 4) % SUBMITTED APPLICATIONS AFFINITY APPLICANT - COMMITTEE

21. Probability to be granted increases with increasing affinity applicants-Committee % GRANTED APPLICATIONS AFFINITY APPLICANT - COMMITTEE

22. Logistic regression analysis:Affinity Applicant-Committee has a significant effect upon the probability to be granted MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 CITATION IMPACT APPLICANT 3 26.97 0.0000 ** Reltransdisc impact applicant 1 0.29 0.5926 AFFINITY APPLICANT-COMMITTEE 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638

23. (viii)Data must be accurate and verifiable

24. (ix)Assessed researchers must have the opportunity to check data and comment on outcomes

25. (x)A framework is needed to characterize and position bibliometric indicators and products

26. Contents

27. The Multi-Dimensional Research Assessment Matrix Expert Group on the Assessment of University-Based Research (AUBR, 2010)

28. Multi‐dimensional Research Assessment Matrix (Part)

29. Multi‐dimensional Research Assessment Matrix (Part) Read column- wise

30. Multi‐dimensional Research Assessment Matrix (Part)

31. Multi‐dimensional Research Assessment Matrix (Part)

32. MD-RAM: Example 1 • Publications in international jrnls; • Actual citation impact • Individual • Hiring/promotion • Productivity & impact • PhD date, place, supervisor; • Invitations for conferences

33. Multi‐dimensional Research Assessment Matrix (Part)

34. MD-RAM: Example 2 • Publications in international jrnls; • (Trend in) actual citation impact • Research group • Monitoring research program • Scientific impact • Collaborations • Topicality

35. Research assessment methodologies: Important considerations • Methodology must befit-for-purpose • What is the primary “problem” to besolved? • Be aware of unintendedeffects • Change a methodologyevery 5-10 years • What is anacceptable “errorrate”? • Wrong inindividualcases  benificiaryfor the system as a whole

36. Contents

37. Journal articles + citations Journal full text data Journal usage data Unit of assess- ment Books Patents Conference Procs Trade jrnls Newspapers Social media

38. (a)Downloads vs. CitationsWhat do full article downloads measure?

39. Authors vs. readers Readers Authors ?

40. Hypothesis on degree of correlation between downloads and citations Authors Authors Readers Readers Strong Weak

41. Usage vs. citations per main field Scientific ? PSYCHOL Societal

42. (b)Patent citations to journal articles:The technological impact of research

43. The Technological Impact of Library Science Research: A Patent Analysis [Halevi et al, 2012] PATENTS (TotalPatent) Citations by patent examiners and inventors 42 LIBRARY SCIENCE JOURNALS (Scopus)

44. Cited articles: keywords in titles The articles feature information retrieval and indexing, information and documents management systems which pertain to electronic and digital libraries development Citing patents: keywords in titles The patents focus on electronic information administration, navigation, and products and services management in commercial systems.

45. (c) International scientific migration

46. International migration vs. co-authorship

47. Map of countries with Ratio migration/collaboration > 1.2

48. Languagesimilarity drives migrationstrongerthanit drives co-authorship Politicaltensions affect migrationlessthanthey affect co-authorship

49. (d)Citation context analysisCombining citation data from Scopus with full text article data from ScienceDirect

50. The use of contextual citations analysis to disclose the thematic and conceptual flow of cross- disciplinary research:the case of the Journal of Informetics 2007 (Gali Halevi et al., to be published, 2012)