1 / 22

Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and Research Proposal

Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and Research Proposal. Asha Balakrishnan Vanessa Peña Bhavya Lal (blal@ida.org), Task Leader November 5, 2011 American Evaluation Association, 2011. Relevant Task Goal.

justin-richard
Download Presentation

Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and Research Proposal

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and Research Proposal Asha Balakrishnan Vanessa Peña Bhavya Lal (blal@ida.org), Task Leader November 5, 2011 American Evaluation Association, 2011

  2. Relevant Task Goal Federal agency wished to know if their set-aside potential transformative research (PTR) program was funding interdisciplinary research (IDR)

  3. Associated Study Questions • How interdisciplinary is the set-aside program’s research [as compared with traditional programs]? • How interdisciplinary is the body of knowledge on which the awarded proposal draws? • How interdisciplinary is the funded PI [as compared with PIs of traditional programs]? • How interdisciplinary is the PI’s prior publication history?

  4. Definition of Interdisciplinarity C.S. Wagner et al. / Journal of Informetrics 165 (2011) 14–26 • Interdisciplinary approaches integrate separate disciplinary data, methods, tools, concepts, and theories in order to create a holistic view or common understanding of a complex issue, question, or problem. • The critical indicators of interdisciplinarity in research include evidence that the integrative synthesis is different from, and greater than, the sum of its parts. • Process of knowledge integration is important

  5. Operationalizing Interdisciplinarity – Concept of Diversity Score • What is the metric? • The “I-score” measures the number of disparate scientific fields connected by a researcher’s body of work. (Porter, 2007) • How is it measured? • By using the subject categories of the cited references. A co-citation analysis of all journals has been measured empirically. • What does it intend to represent? • The I-score intends to measure the interdisciplinarity of the researcher’s work. • Low I-scores indicate that the researcher draws primarily from a specialized body of work.

  6. Evaluative Approach I-Score metric was employed to test if differences existed between test program awards and comparison awards to determine if: Test program is attracting more interdisciplinary PIs Test program is soliciting more interdisciplinary proposals

  7. Comparison Group Selection Selection of Comparison Awards: To assess the difference between the proposals funded by test program and proposals funded through traditional mechanisms, we considered many options and then selected a set of funded proposals from programs as the comparison group. Selection Methodology:

  8. proposal interdisciplinarity

  9. Measuring Interdisciplinarity of Proposals *Porter, et al. Practical research proposal and publication profiling. Research Evaluation 19(1): 29-44 (2010). • Activity: • Gathered the cited references from the Treatment and Comparison Group proposals • Extracted journal titles from references and matched to Thomson ISI subject categories • Used techniques and tools described in Porter et.al., 2010* to compute integration scores

  10. Caveats We assume that the references cited in the awarded proposal represent the interdisciplinarity of the proposed ideas Not all references cited in a proposal were indexed in ISI’s Web of Knowledge database (ranged from 25% to 90% coverage) Subject category assignment of journals is a proxy for “discipline”

  11. Treatment Proposal Integration Scores are Higher than the Comparison Group Proposal Integration Scores

  12. Visual Representation of Average Test and Comparison Group Proposal Integration Score Test Program FY2007-FY2009 • Average I-Score = 0.64 Comparison Group FY2007-FY2009 Average I-Score = 0.55 Clin Med AgriSci EnvSci & Tech Clin Med Chem MatlSci MatlSci Geo Geo Biomed Biomed InfDis InfDis Ecol Sci Ecol Sci Cog Sci Psych Psych Phys Phys Hlth& Soc Eng Eng Bus & Mngt Bus & Mngt Comp Sci Comp Sci Econ. Polit. & Geo Econ. Polit. & Geo

  13. Distribution of Disciplines – Project Level I-Score = 0.64 I-Score = 0.68 I-Score = 0.65

  14. Researcher interdisciplinarity

  15. Measuring Principal Investigator Interdisciplinarity *Porter, et al. Measuring researcher interdisciplinarity. Scientometrics 72(1): 117-147 (2007). • Activity: • Searched and obtain each of the Treatment and Comparison Group Principal Investigator’s publication histories (from Jan 1980 to June 2010) in the ISI Web of KnowledgeSM database • Employed integration score technique based on Porter et.al* to determine integration score of each PI (44 test PIs and 44 Comparison group PIs) • For PIs with a low number of publications, sensitivity analyses were conducted to ensure conclusions still held

  16. Caveats Only journal publications available in ISI’s Web of Knowledge database included Subject category assignment of journals is a proxy for “discipline” Only measured the PI’s interdisciplinarity, not those of the Co-PI’s or the team

  17. No Difference Between Treatment PI Integration Scores and Comparison Group PI Integration Scores

  18. Average Test and Comparison Group PI Integration Score Test Program FY2007-FY2009 Average PI I-Score = 0.50 Comparison Group FY2007-FY2009 • Average PI I-Score = 0.48 Clin Med EnvSci & Tech Clin Med EnvSci & Tech Biomed MatlSci Biomed MatlSci InfDis Chem InfDis Chem Cog Sci Cog Sci Geo Geo Phys Hlth& Soc Eng Eng Bus & Mngt Econ. Polit. & Geo Comp Sci Comp Sci Econ. Polit. & Geo

  19. Summary of Findings I-scores for test group PIs and comparison group PIs are not different I-scores from test groups proposal cited references are higher than I-scores from comparison group proposal cited references Set-aside PTR program is not attracting PIs with a more interdisciplinary background than other programs. However, program is successful attracting researchers to submit more interdisciplinary ideas!

  20. Methodological Finding: I-score does not tell the complete story Example PI Maps Example Proposal Maps I-Score = 0.72 I-Score = 0.42 I-Score = 0.54 I-Score = 0.64 I-Score = 0.41 I-Score = 0.73

  21. Caveats: Study Based on Many Assumptions • I-score is a good measure of interdisciplinarity • Little is known about how subject categories are generated and how static they are over time • Journal coverage in databases is incomplete • Proposal cited references reflect knowledge integration • Rhetorical vs. reward citations

  22. Future Questions of Interest • How interdisciplinary was the team (integrate all PI publications into one set and determine i-score) • Are younger PIs more interdisciplinary? • Testing on larger population of awards • What is the relationship between PI and proposal I-scores? • Exploratory look found none in the test group, and significantly positive for comparison group • What is the relationship between Proposal Integration Score and PTR Score? • Exploratory look found none

More Related