1 / 26

The uses, benefits and pitfalls of bibliometrics

The uses, benefits and pitfalls of bibliometrics. Julia Lane American Institutes for Research. TO Describe Scientific Communication. Outline. Context Scientific Communication Bibliometrics Uses, Benefits and Pitfalls Some other ideas and opportunities What can be done.

delora
Download Presentation

The uses, benefits and pitfalls of bibliometrics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The uses, benefits and pitfalls of bibliometrics Julia Lane American Institutes for Research

  2. TO Describe Scientific Communication

  3. Outline • Context • Scientific Communication • Bibliometrics • Uses, Benefits and Pitfalls • Some other ideas and opportunities • What can be done

  4. Scientific Communication Knowledge • Creation • Transmission • Adoption

  5. Open Access Context • Authors: of such articles, who will see their papers more read, more cited, and better integrated into the structure of science • Academic readers: in general at institutions that cannot afford the journal, or where the journal is out of scope • Researchers: at smaller institutions, where their library cannot afford the journal • Readers: in general, who may be interested in the subject matter • The general public: who will have the opportunity to see what scientific research is about • Taxpayers: who will see the results of the research they pay for • Patients: and those caring for them, who will be able to keep abreast of medical research Source: http://en.wikipedia.org/wiki/Open_access_journal accessed May 26, 2012

  6. Bo-Christer Björk, 2007

  7. Scientific Communication • Is about people

  8. Bibliometrics: Uses • …a set of methods to quantitatively analyze scientific and technological literature..used. • …in library and information sciences • …to explore the impact of • their field, • a set of researchers, or • a particular paper. • …. in quantitative research assessment exercises of academic output which is starting to threaten practice based research • Paraphrased from http://en.wikipedia.org/wiki/Bibliometrics

  9. Bibliometrics • Is about documents

  10. Benefits Benefits • Have focused attention on quantitative measures of impact • Attracted some smart people to think about hard problems • Identified some interesting patterns

  11. Pitfalls(Illustrative not Exhaustive) Scientific validity • Limited in its behavioral micro-foundations • Based on suspect scientific frame (unit of analysis, currency, coverage etc.) • Not generalizable or replicable Inferential validity • Generates spurious results • Subject to misuse and gaming Value for evaluation • Creates perverse incentive structure

  12. Illustrative criticism Evaluators often rely on numerically–based shortcuts drawn from the closely related fields (Hood and Wilson, 2001) of bibliometrics and scientometrics — in particular, Thompson Scientific’s Journal Impact Factor (JIF). However, despite the popularity of this measure, it is slow (Brody and Harnad, 2005); narrow (Anderson, 2009); secretive and irreproducible (Rosner, et al., 2007); open to gaming (Falagas and Alexiou, 2008); and based on journals, not the articles they contain. Priem and Hemminger, 2010

  13. Illustrative Example

  14. Why Getting it Right Matters

  15. Measuring Communication:Some Ideas and Opportunities Focus on people (scientists) • How to start a movement Make use of scientific advances • New theories (graph theory; RCT) • New applications (social networks) • New ways of communicating knowledge • New application (graph oriented databases) • New data (natural language processing; computational linguistics) New opportunities => Potential for new science, new scientific field and theoretically grounded, metrics

  16. Conceptual Framework Source: Ian Foster University of Chicago

  17. Scientific Advances • Graph Theory applied to social networks • E.g. Jason Owen Smith • dyadic measures of the strength of individual ties; • structural measures of cohesion in the overall network; and • node demographic measures that highlight the degree of heterogeneity of academic researchers • Randomized Controlled Trials • E.g John Willinsky • Physicians • Community Health Organizations

  18. New Ways of Communicating Knowledge

  19. Scientometrics 2.0: Toward new metrics of scholarly impact on the social Web by Jason Priem and Bradley M. Hemminger.First Monday, Volume 15, Number 7 - 5 July 2010http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/

  20. New Data • Natural language processing (=> mine actual text and identify topics) • Information extraction to capture data on people and institutions

  21. What can be done • Build on current advances to develop and automatically extract de-duplicated cross-referenced database of • papers (and references) • Topics • People • Grants • publication venues • Institutions • Build a common data infrastructure • STAR METRICS

  22. What can Be Done to create New Science • Agree on handful of people based metrics • Engage social scientists to develop open and transparent data and standardized measures • Use them

  23. References (Plus the New Yorker) • McCabe, Mark J. and Snyder, Christopher M., The Economics of Open-Access Journals (July 9, 2010). Available at SSRN: http://ssrn.com/abstract=914525 or http://dx.doi.org/10.2139/ssrn.914525 • McCabe, Mark J. and Snyder, Christopher M., Did Online Access to Journals Change the Economics Literature? (January 23, 2011). Available at SSRN: http://ssrn.com/abstract=1746243 or http://dx.doi.org/10.2139/ssrn.174624 • Philip M. Davis Open access, readership, citations: a randomized controlled trial of scientific journal publishing FASEB J July 2011 25:2129-2134; published ahead of print March 30, 2011, doi:10.1096/fj.11-183988 • Evans, James and Jacob Reimer (2009) “Open Access and Global Participation in Science,” Science 323: 1025. • Leydesdorff, L. (2008), Caveats for the use of citation indicators in research and journal evaluations. J. Am. Soc. Inf. Sci., 59: 278–287. doi: 10.1002/asi.20743 • Scientometrics 2.0: Toward new metrics of scholarly impact on the social Webby Jason Priem and Bradley M. Hemminger.First Monday, Volume 15, Number 7 - 5 July 2010http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/ • Kaye Husbands-Fealing, Julia Lane, Jack Marburger, Stephanie Shipp, and Bill Valdez The Handbook of Science of Science Policy,, Stanford University Press, 2011. • Julia Lane and Stefano Bertuzzi “Measuring the Results of Science Investments” Science, Volume 331, pages 678-680, February 11, 2011. • Julia Lane “Let’s Make Science Metrics More Scientific” Nature, Volume 464, pages 488–489, March 25, 2010. • Julia Lane“Assessingthe Impact of Science Funding” Science, Volume 324. no. 5932, pp. 1273 – 1275, 5 June 2009.

More Related