“Open access and research evaluation” Thursday 19th October

1. Institute for Research Information and Quality Assurance “Open access and research evaluation” Thursday 19th October 14.15-15.45 Session 4: Quality Assessment Open Scholarship 2006: New Challengesfor Open Access Repositories The University of Glasgow 18-20 October 2006 Stefan Hornbostel iFQ Institut für Forschungsinformation und QualitätssicherungInstitute for Researchinformation and Quality Assurance Godesberger Allee 90 D-53175 Bonn www.forschungsinfo.de Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

2. Relations between OA and research evaluation political / rethorical function to spread OA (to maximize the impact of their articles authors should archive!) OA offers a better way to evaluate research performance?(Compensation of shortcomingsof traditional bibliometric Indicators) OA changes scientific behaviour (more easily accessibility, sooner availability, quick circulation, high quality). OA makes it possible to construct new indicators to measure a completely different dimension (usage, reading, …) Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

3. Relations between OA and research evaluation (1) political / rethorical function to spread OA (to maximize the impact of their articles authors should archive!) Research evaluation practices are used to convince authors of OA advantages (higher Citationrates, higher attention) The analysis covers 1,494 publication venues containing at least 5 online and 5 offline articles. For 90% of venues, online articles are more highly cited on average. On average there are 336% more citations to online articles compared to offline articles. Lawrence, S. "Online or Invisible" Nature 411 (6837): 521, 2001. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

4. „Mandated online RAE CVs Linked to University Eprint Archives“, “The Funding Councils should mandate that ……. the full text of every refereed research paper, publicly self-archived in the university's online Eprint Archive and linked to the CV for online harvesting, scientometric analysis and assessment. This will increase the uptake and impact of UK research output, by increasing its visibility, accessibility and usage, and set an example for the rest of the world that will almost certainly be emulated, in both respects: research assessment and research access.” Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives. Ariadne 35. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

5. “The proportion of articles for which their authors provide OA is likely to increase dramatically now, in part because of the mounting evidence for the impact advantage OA confers.”Brody, T. and Harnad, S. (2004) Comparing the Impact of Open Access (OA) vs. Non-OA Articles in the Same Journals. D-Lib Magazine 10(6). Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

6. For Example: Cream of Science “Researchers are enthusiastic about the initiative. A number of them already refer to www.creamofscience.org on their own websites for a complete overview of their work. They regard Cream of Science as a hallmark of quality.” Dare.net 17.05.2005 • Target • “demonstrat(ing) that scholars are willing to deposit their materials in a repository, thereby also increasing the awareness of other scholars” • Method • “DARE partners selected ten of their prominent scientists and made their complete publication list, with as much full text available as possible” • Selection Criteria • “there was no objective criteria to define 'prominent',…but most of the DARE partner used a formal method; selection by the Executive Board and/or using a letter from the Dean to invite academics to be part of Cream” • Martin Feijen and Annemiek van der Kuil 2005: A Recipe for Cream of Science: Special Content Recruitment for Dutch Institutional Repositories', Ariadne Issue 45. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

7. OA a causal factor ? Kurtz et al. claim that there are (at least) three possible, non-exclusive effects that cause the effect of higher citation rates. Open Access Postulate (OA): Because of free, unrestricted access, papers are read more easily and therefore get cited more frequently Early Access Postulate (EA): Papers offered as e-print are available sooner and therefore gain primacy and additional time in press, and therefore they get cited more often Self-selection Bias Postulate (SB) or (Quality Postulate) : The most important -- and therefore most citable -- papers are posted on the Internet Kurtz, Michael J., Eichhorn, Guenther, Accomazzi, Alberto, Grant, Carolyn, Demleitner, Markus, Henneken, Edwin, Murray, Stephen S. (2005). “The Effect of Use and Access on Citations”, Information Processing and Management, Vol. 41, Issue 6, p. 1395-1402. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

8. Open Access Postulate • no relationship between fulltext arXiv downloads and the number of article citations until an article has been downloaded more than 400 times • consistent with the detailed work of Henk Moed on the journal Tetrahedron Letters, which suggests that a relatively small group of both highly cited and frequently downloaded papers are responsible for the weak, though statistically significant, correlation between downloads and citations Philip M. Davis / Michael J. Fromerth (2006): Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

9. Early Access Postulate Partial regression plot of prepublication days on citation, controlling for the number of days since publication and the effect of the journal. Articles deposited in the arXiv after formal publication are highlighted red. We have failed to find support for the Early View postulate and have provided evidence that many highly-cited articles were deposited into the arXiv after formal publication. Philip M. Davis / Michael J. Fromerth (2006): Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

10. Self-selection Bias Postulate (SB) or Quality Postulate • approximately 15% of the articles in a journal collect 50% of the citations • If authors are indeed depositing their best papers, Open Access citation advantage may be an artifact, not the cause of a citation advantage Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

11. Problems: Open Access (green and gold), Open Archive, Self Archiving may produce different quality distributions J. Testa and M.E. McVeigh. 2004.“The Impact of Open Access Journals.” A Citation Study from Thomson ISI, www.isinet.com/oaj . Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

12. OA offers a better way to evaluate research performance?(Compensation of shortcomings of traditional bibliometric Indicators) “These new possibilities may be interesting seen in combination with the known inherent bias of research evaluation using citation analysis. …..”Tove Faber Frandsen (2006): Open access-based resources in research evaluation The Funding Councils should mandate that … all UK research-active university staff must maintain: .... (II) the full text of every refereed research paper, publicly self-archived in the university's online Eprint Archive ….. This will give the UK Research Assessment Exercise (RAE) far richer, more sensitive and more predictive measures of research productivity and impact, for far less cost and effort (both to the RAE and to the universities preparing their RAE submissions)… Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives. Ariadne 35. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

13. ISI / WOS coverage by main field Quelle: Henk v. Moed, Evaluation of Research Performance and Funding Programme in Social Sciences; Tagung: Norface Workshop on Research Programme Development and Management 7.02.06, Bonn: http://www.norface.org/norface/files/file_20060207014936Henk%20Moed.pdf Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

14. Antelman, Kristin (2004) Do Open Access Articles Have a Greater Research Impact?. College & Research Libraries News 65(5):pp. 372-382. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

15. Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation? Results of a study on publishing habits and information acquisition with regard to open access In electrical engineering, computer science and system engineering, 34 % of the respondents know of this form of publication; the highest percentage here is among computer scientists, at 51 %. In mechanical engineering and production engineering, 31 % of the scientists said they knew about open access journals, compared to just 13 % in the area of heat energy technology and process engineering. In the areas of civil engineering and architecture this type of publication is completely unknown. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

16. In the humanities and social sciences, it is mostly social and behavioural scientists who use the option of secondary open access publication for their academic studies. While they make around 9% of the articles published in journals available for free on the internet, this is done by only 3% of the humanities scholars surveyed. In the engineering sciences it is the computer scientists who stand out, with two-thirds of their articles in conference proceedings, 46 % of their journal articles, almost 42 % of their contributions to edited volumes and nearly 24% of their monographs republished on the internet. Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation? Results of a study on publishing habits and information acquisition with regard to open access Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

17. Deutsche Forschungsgemeinschaft (2005): Publishing Strategies in Transformation? Results of a study on publishing habits and information acquisition with regard to open access Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

18. AUGMENTING INTEROPERABILITY ACROSS SCHOLARLY REPOSITORIES A meeting sponsored and supported by Microsoft, the Andrew W. Mellon Foundation, the Coalition for Networked Information, the Digital Library Federation, and the Joint Information Systems Committee. April 20 – 21, 2006, New York, NY • Richer cross-repository services, that is services that overlay multiple repositories • Cross-repository scholarly communication workflows • Examples of such value-chains include: • The transfer of (parts of) digital objects from digital repositories to parties that provide discovery-oriented services over (parts of) the digital objects. • The transfer of digital objects from digital repositories to parties that provide value-added services over the digital objects. • The introduction of natively machine-readable and machine-actionable bibliographic citations. • The re-use of datasets (from different repositories) as the basis for building a new dataset or for writing a publication. Ithaca, NY and Los Alamos, NM - The Open Archives Initiative (OAI), with the generous support of the Andrew W. Mellon Foundation, announces a new effort as part of its mission to develop and promote interoperability standards that aim to facilitate the efficient dissemination of content. Object Reuse and Exchange (ORE) will develop specifications that allow distributed repositories to exchange information about their constituent digital objects. These specifications will include approaches for representing digital objects and repository services that facilitate access and ingest of these representations. The specifications will enable a new generation of cross-repository services that leverage the intrinsic value of digital objects beyond the borders of hosting repositories. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

19. OA changes scientific behaviour (more easily accessibility, sooner availability , quick circulation, high quality) Scholarly communication has changed rapidly over the last decade as scientists now publish their findings in e.g. open access journals and preprint servers. Bauer & Bakkalbasi (2005) The communication system is changing primarily due to the use of new technology. Friend (2004) Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

20. But: only c. 20% of the number of papers published annually are open access Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

21. Poeschl,U. 2006,Peer Review Revisited, 16.05.2006, Ms. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

22. OA makes it possible to construct new indicators to measure a completly different dimension (usage, reading, …) Web indicators based on usage data are a way to obtain unique information of the utility and scholarly acceptance of OA research output, but these new user oriented web indicators are still in an early experimental phase. The rationale of these approaches is strong but standardization of web metrics is at the beginning. …. there is still much to be done in evaluation of OA research. In addition we need further approaches to build robust measures and metrics especially for OA publications/documents. Following the thesis “science is turning to e-science” … we conclude e-science will need new stable e-indicators or a combination with traditional indicators to appropriately take e-only research output into consideration. Philipp Mayr (2006): Constructing experimental indicators for Open Access documents. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

23. The rendered journal impact rankings were compared to those derived from the Reader Generated Network, and the ISI IF. Our results indicate the following: First, journal relationships in the RGN network seem to be valid and representative of the community whose downloads have shaped the network. Second, structural journal impact metrics derived from the RGN deviate strongly from the ISI IF. Third, the applied structural impact metrics correlate strongly with the ISI IF when calculated over the AGN, indicating they do validly operationalize journal impact, if we honor the assumption that the ISI IF does. Fourth, the AGN and RGN networks overlap to some degree, but exhibit striking differences. Darmoni, Roussel, Benichou, Thirion, and Pinhas (2002) compare journal usage frequency to the ISI IF for a medical DL collection. They define a “Reading Factor” (RF) which consists of the ratio of a particular journal’s download frequency to the total downloads of all journals as recorded in the DL’s logs. The authors report a low and statistically insignificant correlation between the observed RF and the ISI IF for the same set of journals. Johan Bollen / Herbert Van de Sompel / Joan A. Smith / Rick Luce (Los Alamos National Laboratory) 2005: Toward alternative metrics of journal impact:A comparison of download and citation data. Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

24. Standards! OA research evaluation should follow the classical methods of bibliometrics: Measuring Research activity P amount of Publications P% Percentage of worldwide Publications (international comparison) or national Publications (national comparison) MeasuringImpact C amount of citations Pp% percentage of publications not cited until now Cmax Maximum of Citations C/P Citation Rate C/P/FCSm (mean field citation score) Relative Citation Rate (related to field) (C/P)/JCSm (mean journal citation score) Relative Citation Rate (related to journal) Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

25. Thank you! Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

26. Difficulties of OA research evaluation Only fits to publications in traditional databases - especially ISI Difficult for publications which are not covered by ISI Missing subfield information Metadata No standardised metadata No gold standard for the evaluation of open access papers No best practices of hundred percent covering Units Main units for OA research evaluation are inexplicit Different kinds of publications Different motives for OA publishing -> bias! Hornbostel 19.10.2006 Institut für Forschungsinformation und Qualitätssicherung

27. It was found that during the first 3 months after an article is cited, its number of downloads increased 25% compared to what one would expect this number to be if the article had not been cited. Moreover, more downloads of citing documents led to more downloads of the cited article through the citation. An analysis of 1,190 papers in the journal during a time interval of 2 years after publication date revealed that there is about one citation for every 100 downloads. A Spearman rank correlation coefficient of 0.22 was found between the number of times an article was downloaded and its citation rate recorded in the SCI. When initial downloads - defined as downloads made during the first 3 months after publication - were discarded, the correlation raised to 0.35. However, both outcomes measure the joint effect of downloads upon citation and that of citation upon downloads. Correlating initial downloads to later citation counts, the correlation coefficient drops to 0.11. Findings suggest that initial downloads and citations relate to distinct phases in the process of collecting and processing relevant scientific information that eventually leads to the publication of a journal article. Hornbostel 19.20.2006 Institut für Forschungsinformation und Qualitätssicherung

28. Preprint submitted to Elsevier Science 17 May 2006 Similarly, Darmoni, Roussel, Benichou, Thirion, and Pinhas (2002) compare journal usage frequency to the ISI IF for a medical DL collection. They define a “Reading Factor” (RF) which consists of the ratio of a particular journal’s download frequency to the total downloads of all journals as recorded in the DL’s logs. The authors report a low and statistically insignificant correlation between the observed RF and the ISI IF for the same set of journals. These results show that journal download frequency within a local DL community does not correspond to the ISI IF, which raises questions regarding the ISI IF’s validity as the sole indicator of Ig among a specific community of readers. 4 These results furthermore raise questions regarding the validity of the ISI IF as the sole assessment of journal impact, and suggest the possibility of devising impact metrics based on usage information in general. Toward alternative metrics of journal impact: A comparison of download and citation data. Johan Bollen Department of Computer Science, Old Dominion University, 4700 Elkhorn Ave., Norfolk VA 23529 Herbert Van de Sompel Research Library, Los Alamos National Laboratory, Los Alamos, NM, 87554 Joan A. Smith Department of Computer Science, Old Dominion University, 4700 Elkhorn Ave., Norfolk VA 23529 Rick Luce Research Library, Los Alamos National Laboratory, Los Alamos, NM, 87554 Hornbostel 19.20.2006 Institut für Forschungsinformation und Qualitätssicherung