Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool.

1. Bibliometric analyses of publications and grant productivity: SPIRES, a new Web-based tool. The National Institute of Environmental Health Sciences The National Institutes of Health • Today’s discussion points: • Background • - Program Analysis Branch Goals • - Current methods for evaluating research including bibliometric analysis • Introduction to SPIRES (Scientific Publications Information REtrieval System) • Results of a Validation Experiment • Future Directions/ • Live demo

2. PAB Analysis Goals • Information transfer– Communicate the high-quality research produced by the NIEHS portfolio. • Track current DERT supported research– Examine number of grants and annual support broken down into major fields of science. • Analyze all incoming grants– Develop a two-three sentence description of each funded grant, place into a web searchable data base. Track study sections and priority scores. • Evaluate scientific and public health impact of grants.Track publications for each grant award and center. Every RFA and PA needs to be tracked for publication productivity and scientific impact.

3. Evaluation Tools • peer review– accepted “gold standard of evaluation,” provides a review of quality of research • – usually applies to specific projects, difficult to use for the evaluation of programs or fields, time consuming, expensive and labor intensive • bibliometric analysis– examines publications, impact factors, literature citations, and patent counts; is based on the premise that a publication represents quality research as it passed peer review, and cited literature has merit • – quantitative can measure aggregate quality of fields or programs, but not always useful across fields

4. Needs for tracking publications • To systematically track the productivity of the portfolio, specific tools are needed by the Program Analysis Branch (PAB) that are not currently available. • According to NLM, NIEHS-supported grantees published 1637 publications in 2000. • The idea was to link the grant database (IMPAC II) to the National Library of Medicine (NLM) database. • Web-based routines will allow NIEHS staff to display information and generate multiple outputs. • System should be robust so that new NLM downloads can be accepted and operator updates are feasible.

5. Development of SPIRES: joining IMPAC II with NLM. • SPIRES :Scientific Publication Information REtrieval System • NLM keeps all its data in flat files. • Jerry Nehls downloaded ES/NIEHS data from NLM to a local server. • Paul Jordan developed computer algorithms to search publications for Grant numbers, (ES-----) and developed tables in Oracle to create a relational database linking IMPAC II with NLM data. • Bob Hoppin developed a Web-based interface. • Paul Jordan and Ying Gao currently upgrading functionality and usability.

6. Status of SPIRES.Scientific Publication Information REtrieval System • Currently have 1995-2001 (August) data loaded with 9,615 publications • SPIRES is being used by the PAB: – Finding and removing bugs – Working on developing a better interface and reportingstructure • SPIRES is accessible to anyone in NIEHS http://carlton.niehs.nih.gov/SPIRES/PRIV/query/query_sel.cfm • Working on the ambiguous Project code problem. – ???ES007218 – rarely is the prefix captured: R01, T32, F32, P01 – ES-007218 – hyphen present or zero missing?

7. Status of SPIRES (contin.).Scientific Publication Information REtrieval System • Implement and expand system through theSPIRES Working Group: PAB, Computer Applications Specialists, DERT Staff, Sheila Newton, Larry Wright • Running test reports: looking at data quality through validation experiments • SPIRES as a tool for analysis– develop reports on Arsenic, Oxidative Stress, Cardiovascular disease • Quarterly updates to SPIRES • V 1.0 was beta-tested by several groups at NIH • V 2.0 being developed and tested http://carlton.niehs.nih.gov/SPIRES/PRIV/query/query_sel.cfm V2.0: https://dobby.niehs.nih.gov/spires/priv/main/mainmenu.cfm

9. Searching for pub information Searching for grant information Crisp terms

10. Searched: oxidative stress, and reactive oxygen Crisp terms

19. Impact factor = total citations/journal articles/yr

23. Total pubs = 23, since 1995

25. Searched projects for : oxidative stress, and reactive oxygen Crisp terms

26. Crisp terms

27. Projects

38. Publications from 1995- August of 2001- sorted by publication count

39. SPIRES Validation – Jerry Phelps • Compare SPIRES data to information contained in the grant file (progress reports and competitive renewals) • Grants Years • – Pilot studies : MERIT n=16 1995-2001 • R03 n= 16(44) 1995-2001 • – Larger study: R01 n=20 1995-2001 P01 n=2 1995-2001 • P30 n=3 2000 • P42 n=3 2000

40. MERIT R03 R01 P01 P30 P42 COMBINED GRANT FILES 235 29 329 69 275 49 986 SPIRES 202 86% 25 86% 248 75% 44 69% 95 35% 28 57% 642 65% SPIRES Validation Pilot studies Larger study PAPERS “R” AWARDS: 475/593 “P” AWARDS: 167/393 80% 47%

41. SPIRES Validation • Based on these validation studies, Paul Jordan and Jerry Nehls have made improvements in the download process and decoding algorithms that will improve SPIRES’ ability to capture papers from PUBMED. • PAB is following up with an analysis of the entire P30 program to give the Center Directors data that they will use to increase the number of papers that cite Center support. • DERT and PAB will be contacting PI’s who are not citing their grants in their publications.

42. SPIRES as an Analysis Tool 1. Comparisons of productivity for MERIT nominees. 2. Productivity assessments for competitive renewal applications. 3. Identify high-impact papers from grantees who are visiting NIEHS or being visited by NIEHS leadership. 4. Program Evaluations – Productivity assessments for grant programs such as arsenic, oxidative stress, P01, Children's centers, and R03. 5. Customized reports in Excel spreadsheets allow for easy sorting according to impact factor, date, journal, PI, institution.

43. SPIRES – Future Directions • Develop better reporting features • Integrate with Centers, Superfund, Contracts, and DIR databases • Incorporate links to PubMed and ISI Citation reports • Demo to other IC’s, working towards building a NIH-wide system • Consider incorporating SPIRES into the Quick View Reports (QVR) section of the Electronic Council Book • Add additional tracking fields (through QVR): grant descriptors, disease, chemical, and organ systems • Consider making a site available on Web for grantees

44. Special thanks to: Paul Jordan, Ying Gao, Bob Hoppin & Jerry Nehls (CTB) Nancy Stegman (CTB) Ben Van Houten, Jerry Phelps, Martha Barnes, Edith Lee, Judy Hanson (PAB) Thor Fjellstedt, Bill Suk, & Anne Sassaman (DERT) Larry Wright (DIR, Library) Sheila Newton (OPPE)