Bibliometric Analysis Tools for Research Portfolio Analysis and Management

1. Bibliometric Analysis Tools for Research Portfolio Analysis and Management Ann Kushmerick Manager, Research Evaluation and Bibliometric Data May 3, 2013

2. Thomson Reuters • LARGEST PROVIDER OF INTELLIGENT INFORMATION • Thomson Reuters is the largest provider of intelligent information to business and professional customers in the world. GLOBAL OBJECTIVE EXPERIENCED We operate in over 90 countries around the world. We support over 5,600 universities, governments and research institutions from more than100 countries. We offer unbiased metrics based on citation activity of the most impactful global and regional journals to the scholarly community, and remain free from proprietary involvement. Our roots run deeper than any other information partner today. And our research evaluation experts have demonstrated experience in science policy, research administration and project management.

3. Thomson Reuters: Solutions for the Scientific Research Community

4. Solutions for Research Funders • Connecting research to impact for the world’s leading research funding organizations, governments, publishers, associations and institutions… Benchmarking and comparison products Journal evaluation and comparisons tools • Customized research impact evaluations • Expert services Customized grant management reporting • Customized research portfolio analysis RESEARCH PRODUCING ORGANIZATIONS INTELLIGENT INFORMATION ADVANCED TOOLS AND ANALYTICS PROFESSIONAL SERVICES

5. Critical Decision Support for Research Funders • Benchmark and compare performance and output to peers • Identify and promote partnerships or collaborations both inside and outside the organization • Identify and promote areas of strength or specializations tied to impact • Decision support for policy and program development • Comprehend the research landscape • Provide your teams with flexible, customized systems • Position with board members and key stakeholders • Identify emerging subject areas or researchers • Demonstrate results and impact of research activities • Identify new trends and key indicators to enable policy development

6. INTELLIGENT INFORMATION

7. Web Of Science: The foundational data for research analytics solutions Selectivity and control of content:consistent, transparent standards 12,000 prestigious journals and 716 million+ Cited References Multidisciplinary: Sciences, Social Sciences, Arts & Humanities Depth of more than 100 years, including cited references Consistent and reliable data: • Cover to cover indexing • Cited references for all years of coverage • Indexing of all author names and addresses Unmatched expertise: 50+ yrs. of citation indexing and analysis Conferences and Scholarly Books: complete the research picture Funding acknowledgments: link funding source to outputs The gold standard used by 5,000+ institutions in over 100 countries

8. Datasets to enable your analysis • We offer customized datasets and studies from any of our data sources: • Web of Science- (data can be customized or full database) • Medline • Journal Citation Reports • Essential Science Indicators • InCites • ScienceWire: Our data warehouse where funding inputs (grants) and outputs (journal articles, conference proceedings, patents, etc.) are linked.

9. Expert staff

10. ABOUT OUR RESEARCH ANALYTICS CAPABILITIES Scalable, global team of experts with strong research backgrounds • Highly Trained: Our global product managers, consultants, account managers and customer training teams are credentialed in all major fields and are experts in evaluation, bibliometrics, and management in Library Science, Bioinformatics, Chemistry, Neuroscience, Physics, Statistics, Mathematics, Science Policy and Evaluation. • Highly Experienced: We have demonstrated expertise in science policy, research administration and project management with an understanding and appreciation of the scientific discovery process. And because we have teams around the world, we are able to provide evaluations and objective reporting in global and regional contexts.

11. Tailored solutions

12. PROVIDING UNMATCHED CONTENT, TOOLS, AND SERVICES • Providing a clear picture of research activities and offering decision support to customers around the world - from answering a single question to creating an enterprise-wide decision support system BENCHMARK PROFILE CONSULT

13. InCites Bibliometric analysis of research output and impact with peer comparisons. 13

14. Questions InCites answers for funders What papers resulted from our funding? Which have the highest impact in their field? Which of our funded authors are producing high impact research? What fields of research are they publishing in? What journals? What institutions are we funding? How do they compare to each other? What authors, institutions, and countries are citing our funded works?

15. InCites offers different types of analysis to answer a variety of research evaluation questions • Research Performance Profile: Run custom reports on journal and conference literature you have funded. Measure researcher impact, collaboration success, and field strengths. Global Comparisons: Compare research impact of countries and universities. Analyze trends back to 1981. Institutional Profiles: Compare institutions using a 360o view. Includes bibliometric indicators, academic reputation survey, and institution submitted data,

16. Normalized citation metrics put citation information in context • Citation rates vary among fields. What is good or average in mathematics is very different from what is good or average in biochemistry. • 23.3 cites/paper • H-index: 13 • 14.5 cites/paper • H-index: 7 • 9.8 cites/paper • H-index: 7 • 4.2 cites/paper • H-index: 3 • How “good” is this? What is the context? • Additional metrics are needed to understand research performance.

17. A VARIETY OF INDICATORS

18. Guidelines for citation analysis For further guidance on citation metrics, download the white papers at: http://science.thomsonreuters.com/info/bibliometrics/http://isiwebofknowledge.com/media/pdf/UsingBibliometricsinEval_WP.pdf • Compare like with like – The Golden Rule • Use relative measures, not just absolute counts • More applicable to hard sciences than arts/humanities • Know your data parameters: • journal categories • author names • author addresses • time periods • document types • Obtain multiple measures • Recognize skewed nature of citation data • Ask whether the results are reasonable • Understand your data source

19. InCites: Global ComparisonsAnalyze global trends in publication Contains data on: -countries and regions -9 category schemes -thousands of institutions

20. Analyze Research Activity by Country Public, Environmental, and Occupational Health is a growing field in these Asian countries, particularly China.

21. Analyze Research Activity by Country For these countries, relative citation impact in this field is close to the world average. 1.0 represents the world average (field baseline)

22. Identify which institutions are contributing to China’s growth These Chinese institutions are producing the most research in this field.

23. InCites: Research Performance Profile custom built for detailed analysis of your publications

24. Options for building a Research Performance Profile for a Funder Web of Science funding acknowledgements Medline funding acknowledgments for records shared by Medline and Web of Science Matching data provided by the customer: Pubmed IDs, UTs (Web of Science unique identifier), bibliographic records supplied by funder

25. InCites includes a detailed dataset built to your specifications View detailed metrics on each article from your dataset This paper ranks in the top 0.23% of all neuroscience reviews from 2000. Compare a citation count to journal and category norms. This paper clearly outperforms the average. Examples taken from Mt Sinai School of Medicine sample dataset

26. Analyze researcher performance: Neurosciences Sort by number of documents to identify the most prolific authors These metrics are unique to InCites. They normalize citation performance by research field. Hof, PR received over 2 times the expected rate of citation in neuroscience, and on average his papers rank in the top 18% of neuroscience papers.

27. Generate dashboards for a variety of citation metrics, e.g. Hof, PR Mean percentile: on average, Hof’s papers rank in the top 20% of their respective fields Journal actual/expected ratio: Hof gets cited 1.4 times the average for the journals he publishes in H index: 65 papers have been cited at least 65 times each Self citation analysis: 13% of Hof’s citations are self citations. H index without self citations is 58.

28. Hof, PR: analyze citation growth year on year

29. Identify potential new areas of excellence: analyze data by research field Medical ethics comprises a small portion of Mt Sinai’s total output, but the category normalized impact is over 2 times the expected rate.

30. Understand research collaborations among your funded institutions Mt Sinai doesn’t collaborate often with McGill Univ, but the research has high impact in its field.

31. What is the global impact of your research? Identify government bodies, global universities, and corporations who are building on your work.

32. Understand the co-funders of your research Where else are your fundees getting support? Look for specific grant numbers. View the topics funded by Dept of Veterans Affairs Drill down to papers funded by Pfizer.

33. Save time when creating reports; efficiently support evidence-based decision making • Save your custom reports and share with other users • Export reports to PDF and Excel • Save and share your report parameters with other users

34. Custom projects Our experts can help you answer your key evaluation questions.

35. Electronic Scientific Portfolio Assistant (eSPA): analyzes linkages from grants to their measurable outputs 36

36. The Global Landscape of Autism Research HHS IACC/OARC Autism Spectrum Disorder: The Global Landscape of Autism Research http://iacc.hhs.gov/publications-analysis/july2012/index.shtml How does the volume of autism publications align with a strategic plan advancing autism research? Automated classification of >10,000 publications with 85% accuracy.

37. The Global Landscape of Autism Research Share of ASD publications globally. “All Other Countries” category is led by China, Taiwan, South Korea, India, Brazil, and Poland The US accounts for the largest number of ASD publications each year The contribution from other countries is growing over time

38. Case Study: Alzheimer’s Research UK • A UK-based biomedical charity was concerned about research capacity in dementia and wanted to use publication data to investigate the statement “research capacity in dementia is lower than in other diseases such as cancer or heart disease” http://www.alzheimersresearchuk.org/siteFiles/resources/documents/reports/ARUK_Defeating_Dementia_-_Building_capacity_to_capitalise_on_the_UKs_research_strengths.pdf

39. Global Research Report: Neglected Tropical Diseases Download free Global Research Reports: http://sciencewatch.com/global-research-reports

40. Capture Progress Reports Real-time, Dynamic Progress Report Evaluation System (NCI-iTRAQR) Center A, June 2012

41. Capture Progress Reports Real-time, Dynamic Progress Report Evaluation System (NCI-iTRAQR) 2007 2004 2005 2006 • Build grant progress report data model, database, and data entry tool • Improved data collection on progress report forms • Use bibliometrics to identify research areas, collaboration, and merging of fields. • Examine research collaborations using co-authorships and network graphs 2010 2009 2008

42. Patent Tracking non-publication outcomes:How has the National Cancer Institute supported development of breast cancer biomarkers? Indication Biomarker • Integrity Biomarker data includes references to Web of Science publications and US PTO patents. • Patents and Publications acknowledge funding from NCI. Integrity Biomarkers Data Validity Status Acknowledgement of NCI Support NCI Grant Publication

43. Tracking non-publication outcomes: How has the National Cancer Institute supported development of FDA approved Drugs? NCI Grant Publication NDA-Listed Patent [USPTO Patent] Approved Drugs [FDA Orange Book] • FDA products linked to NCI funding through acknowledgment in patents listed in New Drug Applications (NDA). • Direct patent acknowledgements represent direct links to the NCI projects. • Additional candidate links established through non-patent references found in the patents listed in the NDA. Non-Patent Reference List Patent Acknowledgement of NCI Support Acknowledgement of NCI Support

44. CUSTOM INDICATORS AND EVALUATIONSBreakthrough Indicator- Our goal is to create a Breakthrough Paper Indicator as a multidimensional vector • Multi-dimensional vector for predicting breakthroughs based on publication characteristics such as: • Citation rates • Geographical patterns of references and citations • Disciplinary patterns of references and citations

45. Innovative Medicines Initiative Joint Undertaking (IMI) Evaluation The Innovative Medicines Initiative Joint Undertaking (IMI) is a public private partnership between the European Union and the European Federations of Pharmaceutical Industries and Associations (EFPIA). With a €2 billion euro budget, IMI supports collaborative research projects and builds networks of industrial and academic experts in Europe that will boost innovation in healthcare. The first funding call to researchers was announced in 2008

46. Innovative Medicines Initiative Joint Undertaking (IMI) Evaluation IMI wanted to evaluated its funding by: • Identifying publications • Citation metrics and analysis • Innovation and relevance to new technologies • Network analysis • Journal and patent analysis First report (October2012): http://www.imi.europa.eu/sites/default/files/uploads/documents/IMI%20JU%20Bibliometric%20Report%201_FINAL.pdf Second report (March 2013):  http://www.imi.europa.eu/sites/default/files/uploads/documents/IMI_BibliometricReport_2_final.pdf

47. Innovative Medicines Initiative Joint Undertaking (IMI) Evaluation

48. Past performance

49. Thomson Reuters Custom Analytics: Used Across The Globe For Evaluation And Policy

50. Future vision