1 / 52

Stevan Harnad EuroCris Evaluation of Research using a CRIS 2010 Bruxelles Palais des académies

Evaluation of Research using a CRIS… and an Open Access Repository and Open Access Metrics and an Open Access Self-Archiving Mandate . Stevan Harnad EuroCris Evaluation of Research using a CRIS 2010 Bruxelles Palais des académies. Repository, CRIS, Mandate, Metrics. Collaborators:.

gallia
Download Presentation

Stevan Harnad EuroCris Evaluation of Research using a CRIS 2010 Bruxelles Palais des académies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of Research using a CRIS…and an Open Access Repository and Open Access Metrics and an Open Access Self-Archiving Mandate Stevan Harnad EuroCris Evaluation of Research using a CRIS 2010 Bruxelles Palais des académies

  2. Repository, CRIS, Mandate, Metrics EuroCris Sept 13 2010

  3. Collaborators: • Brody, Tim (U. Southampton, Eprints) • Carr, Les (U. Southampton, EPrints) • Gargouri, Yassine (U. Québec à Montréal) • Gingras, Yves (U. Québec à Montréal) • Gutteridge, Chris (U. Southampton, Eprints) • Hajjem, Chawki (U. Québec à Montréal) • Harnad,Stevan (U. Québec à Montréal, U. Southampton, EPrints) • Hitchcock, Steve (U. Southampton, EPrints) • Jeffery,Keith (STFC, EuroCris, Cerif) • Larivière,Vincent (U. Québec à Montréal) • Oppenheim, Charles (U. Loughborough) • Sale, Arthur (U. Tasmania) • Swan, Alma (U. Southampton, EPrints, Key Perspectives) EuroCris Sept 13 2010

  4. Overview • Open Access: What? • Open Access: How? • Open Access Mandates • Open Access: Why? • Open Access Advantage • Open Access Artifact? • Open Access Metrics • Open Access: How? EuroCris Sept 13 2010

  5. 1. Open Access: What? EuroCris Sept 13 2010

  6. Open Access is: • Free, • Immediate • Permanent • Full-Text • On-Line • Access EuroCris Sept 13 2010

  7. Open Access to What? • 1. Books • 2. Textbooks • 3. Magazine articles • 4. Newspaper articles • 5. Music • 6. Video • 7. Software • 8. “Knowledge” • 9. Data • 10. Unrefereed Preprints ~2.5 million articles yearly ~25,000 peer-reviewed journals EuroCris Sept 13 2010

  8. 2. Open Access: How? EuroCris Sept 13 2010

  9. Two ways to provide OA: Green OA Self-Archiving Gold OA Publishing EuroCris Sept 13 2010

  10. New impact cycles:New research builds on existing research Impact cycle begins: Research is done Researchers write pre-refereeing “Pre-Print” 12-18 Months Submitted to Journal Pre-Print reviewed by Peer Experts – “Peer-Review” Pre-Print revised by article’s Authors Refereed “Post-Print” Accepted, Certified, Published by Journal Researchers can access the Post-Print if their university has a subscription to the Journal EuroCris Sept 13 2010

  11. More impact cycles: Impact cycle begins: Research is done Researchers write pre-refereeing “Pre-Print” Submitted to Journal 12-18 Months Pre-Print reviewed by Peer Experts – “Peer-Review” Pre-Print revised by article’s Authors Refereed “Post-Print” Accepted, Certified, Published by Journal Researchers can access the Post-Print if their university has a subscription to the Journal New impact cycles: New research builds on existing research EuroCris Sept 13 2010

  12. 3. Open Access Mandates EuroCris Sept 13 2010

  13. There are plenty of repositories EuroCris Sept 13 2010

  14. But almost all of them are almost-empty of OA’s target content (5-25%) 2002-2006 2002-2008 EuroCris Sept 13 2010

  15. Estimates of Current Proportion of Articles that are Open Access (Green & Gold) Björk et al 2010 Björk et al 2010 Gargouri et al 2010 EuroCris Sept 13 2010

  16. Author Surveys (Alma Swan) EuroCris Sept 13 2010

  17. EuroCris Sept 13 2010

  18. Open Access: How? By mandating Green OA Self-Archiving Brody et al (2007) Incentivizing the Open Access Research Web: Publication-, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3). http://eprints.ecs.soton.ac.uk/14418/ OA Metrics motivate OA Mandates And OA Mandates maximize OA Metrics EuroCris Sept 13 2010

  19. Optimal OA Mandate • “I wish to remind you that, as announced a year ago in March 2007, starting October 1st, 2009, only those references introduced in ORBi will be taken into consideration as the official list of publications accompanying any curriculum vitae for all evaluation procedures 'in house' (designations, promotions, grant applications, etc.). “ – Bernard Rentier, Recteur, Université de Liège EuroCris Sept 13 2010

  20. 4. Open Access: Why? EuroCris Sept 13 2010

  21. Open Access: Why? • To maximise the uptake, usage, applications and impact of the research output of your university • To measure and reward the uptake, usage, applications and impact of the research output of your university (research metrics) • To collect, manage and showcase a permanent record of the research output and impact of your university Copenhagen Sep 3

  22. “Online or Invisible?” (Lawrence 2001) “average of 336% more citations to online articles compared to offline articles published in the same venue” Lawrence, S. (2001) Free online availability substantially increases a paper's impact Nature 411 (6837): 521. http://www.neci.nec.com/~lawrence/papers/online-nature01/ EuroCris Sept 13 2010

  23. Lawrence (2001) findings for computer science conference papers. More OA every year for all citation levels; higher with higher citation levels EuroCris Sept 13 2010

  24. 5. Open Access Advantage EuroCris Sept 13 2010

  25. EuroCris Sept 13 2010

  26. Early Access Advantage: OA is accelerating the research access/usage/citation cycle. OA articles are being cited sooner and sooner (Data from Physics Arxiv) EuroCris Sept 13 2010

  27. Cumulative citations 1998-2001 1999-2008 2000-2008 2001-2008 2002-2008 2003-2008 2004-2008 2005-2008 2006-2008 EuroCris Sept 13 2010

  28. Data from arXiv Downloads (“hits”) in the first 6 months correlate with citations 2 years later Most articles are not cited at all Earlier download metrics correlated with later citation metrics Brody, T., Harnad, S. and Carr, L. (2006) Earlier Web Usage Statistics as Predictors of Later Citation Impact. Journal of the American Association for Information Science and Technology (JASIST) 57(8): 1060-1072. http://eprints.ecs.soton.ac.uk/10713/ EuroCris Sept 13 2010

  29. 6. Open Access Artifact? EuroCris Sept 13 2010

  30. Davis et al’s test of self-selection hypothesis OA OA Advantage by randomizing OA EuroCris Sept 13 2010

  31. 2002-2006 2002-2008 EuroCris Sept 13 2010

  32. EuroCris Sept 13 2010

  33. Proportion articles at each citation count level Breakdown by OA and non-OA Mandated Self-Selected 2002 2003 2004 2005 2006 EuroCris Sept 13 2010

  34. Engineering OA/NOA 128/992 Biology 225/865 Biology Research 82/1039 Chemistry 38/1082 Psychology 275/839 Mathematics 360/690 Clinical Medicine 26/1094 Health 841/267 Physics 267/853 Social Science 233/871 Earth Science 265/855 EuroCris Sept 13 2010

  35. Comparing the self-selected OA Advantage With the mandated OA Advantage EuroCris Sept 13 2010

  36. The 15 predictor variables used in the regression to predict citation counts EuroCris Sept 13 2010

  37. Multiple regression by citation ranges EuroCris Sept 13 2010

  38. 7. Open Access Metrics EuroCris Sept 13 2010

  39. EuroCris Sept 13 2010

  40. EuroCris Sept 13 2010

  41. Time-Course and cycle of Citations (red)and Usage (hits, green)Witten, Edward (1998) String Theory and Noncommutative Geometry Adv. Theor. Math. Phys. 2 : 253 Preprint or Postprint appears. 2. It is downloaded (and sometimes read). 3. Next, citations may follow (for more important papers)… 4. This generates more downloads… 5. More citations... EuroCris Sept 13 2010

  42. RAE 2001 Rankings for Psychology EuroCris Sept 13 2010

  43. Ordinary 2-variable linear regression (correlation): one target (Y), one predictor (X) Y = rX + e Multiple regression: one target (Y), several predictors (X1, X2… Xn) Y = β0 + β1X1 + β2x2… + βnXn + e (Logistic regression: curve fitting for non-normal distributions) EuroCris Sept 13 2010

  44. Research Assessment, Research Funding, and Citation Impact “Correlation between RAE ratings and mean departmental citations +0.91 (1996) +0.86 (2001) (Psychology)” “RAE and citation counting measure broadly the same thing” “Citation counting is both more cost-effective and more transparent” (Eysenck & Smith 2002) http://psyserver.pc.rhbnc.ac.uk/citations.pdf EuroCris Sept 13 2010

  45. Contributors to the OA AdvantageEA + QA + UA + (CA) + (QB) • EA: Early Advantage: Self-archiving preprints before publication hastens and increases citations (higher-quality articles benefit more: top 20% of articles receive 80% of citations) • QA: Quality Advantage: Self-archiving postprints immediately upon publication hastens and increases citations (higher-quality articles benefit more) • UA: Usage Advantage: Self-archiving increases downloads (higher-quality articles benefit more) • (CA: Competitive Advantage):OA/non-OA advantage (CA disappears at 100%OA, but very important today!) • (QB: Quality Bias):Higher-quality articles are self-selectively self-archived more (QB disappears at 100%OA) EuroCris Sept 13 2010

  46. EuroCris Sept 13 2010

  47. Sample citation and download growth with time. (Downloads only start in 2005 because that is when this paper was deposited.) Early growth rate and late decay metrics for downloads and citations can also be derived. EuroCris Sept 13 2010

  48. Citations (C) CiteRank (like Google) Co-citations Downloads (D) C/D Correlations Hub/Authority index Chronometrics: Latency/Longevity Endogamy/Exogamy Book citation index Links Tags Commentaries Journal Impact Factor Sample of candidate OA-era metrics: • h-index (and variants) • Co-authorships • Publication counts • Number of publishing years • Semiometrics (latent semantic indexing, text overlap, etc.) • Research funding • Students • Prizes EuroCris Sept 13 2010

  49. 8. Open Access: How? EuroCris Sept 13 2010

  50. Open Access: How? By mandating Green OA Self-Archiving Brody et al (2007) Incentivizing the Open Access Research Web: Publication-, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3). http://eprints.ecs.soton.ac.uk/14418/ OA Metrics motivate OA Mandates And OA Mandates maximize OA Metrics EuroCris Sept 13 2010

More Related