Assessment of Defense Experimental Program: Results and Recommendations

Results of the Assessment of the Defense Experimental Program to Stimulate Competitive Research (DEPSCoR) Brian Zuckerman Presented to COSEPUP Evaluation of the National Science Foundation's Experimental Program to Stimulate Competitive Research (EPSCoR) and Similar Programs in Other Federal Agencies December 17th, 2012

Study Origin and Timeline • FY2008 Defense Authorization Act (Public Law 110-181), Section 241, instructed the Secretary of Defense to utilize a defense Federally Funded Research and Development Center (FFRDC) to carry out an assessment of the DEPSCoR program. • Institute for Defense Analyses was tasked to conduct study February 2008 • Results briefed to Senate and House Armed Services Committee staff (SASC/HASC) November 2008 • Study results cleared for public distribution January 2009

Legislative Mandate for Study • Tangible results and progress toward the objectives of the program • Applications used by, or supportive of, operational users • Expanded national research infrastructure • Activities consistent with statute • Assessment of program elements • Assessment of activities of state committees • Advantages and disadvantages of institution & state based formulas • Mechanisms for improving the management and implementation of the program

Descriptive Statistics • Between 1993 and 2008, 729 total DEPSCoR awards • 546 individual Principal Investigators (PIs) • 121 PIs with multiple awards • 42 with three or more • 1 PI with eight awards • 22% of PIs have won 42% of awards • 1993-2008 funding of $243 million • Decline after 2000 peak partially reversed in 2008 competition • 27 states and territories (states) have been eligible for at least one year since program authorized in current form in 1995, plus Missouri (eligible in 1993) • All eligible states except for the Virgin Islands have won awards • 19% (5) of eligible states have won 35% of awards • Montana, Alabama, Oklahoma, Nebraska, South Carolina • 7% (5) of institutions have won 28% of awards • Montana State, U. Nebraska-Lincoln, U. Wyoming, West Virginia U., U. Arkansas

DEPSCoR Program Objectives To enhance the capabilities of institutions of higher education in eligible states to develop, plan, and execute science and engineering [S&E] research that is competitive under the peer-review systems used for awarding federal research assistance To increase the probability of long-term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government for science and engineering research

The DEPSCoR State Share of DOD S&E Increased Source: National Science Foundation (NSF) Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year Note: Graph includes all states ever involved in DEPSCoR program 1995-2008 Note: The dotted lines represent linear regression models applied to the data

DEPSCoR Funding As a Percentage of DOD Funding in DEPSCoR States • DEPSCoR has declined in importance as a source of funding for eligible states since 2000 Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year

DEPSCoR-eligible states fell into four groups: 6 states Near or above threshold (AL, HI, LA, MS, NM, SC) 9 states Rising fast (AK, ID, KY, ME, MT, NE, NV, ND, SD) 6 states Middle (AR, DE, KS, OK, RI, TN) 4 states + 2 territories Lagging (NH, PR, VT, VI, WV, WY) “Success” Varied Among States Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR database, and the DEPSCoR program database of funding totals by state and year

Average DEPSCoR Funding as a Fraction of DOD Funding by State • DEPSCoR share of university S&E R&D funding varies substantially by state • More than 60% for VT, WY • “Rising fast” states’ (AK, ID, KY, ME, MT, NE, NV, ND, SD) DEPSCoR shares decline 2001-2005 compared with 1993-2000 Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR database, and the DEPSCoR program database of funding totals by state and year Note: Average calculated as total DEPSCoR funds during eligible years divided by total DOD funds during eligible years

Expanded National Research Infrastructure (1) • Involving new investigators: • Recent (2006-2008) cohorts had about 60% new PIs • Most Army-funded DEPSCoR awardees (82%) had not previously received funding from Army Research Office (ARO) • 56% of PIs had been funded by the NSF either previous to or within the same year of their first DEPSCoR award • Training graduate students and postdoctoral fellows: • ARO and Office of Naval Research (ONR) data suggest that awards fund about 1 PhD, 1 Master’s degree, 2 postdocs • Building physical infrastructure: • DEPSCoR awards have supported purchase and maintenance of equipment but data not collected systematically by services

Expanded National Research Infrastructure (2) • Leveraging new funding for defense-related research is limited: • 8% of non-DEPSCoR ARO awardees in DEPSCoR states received a DEPSCoR award before (or in the same year as) their first non-DEPSCoR ARO award • 4 DEPSCoR awardees (less than 1%) won a DOD Multidisciplinary University Research Initiative (MURI) award after or in the same year as their first DEPSCoR award • 2 of these investigators received non-DEPSCoR DOD funding before their first DEPSCoR award • Leveraging other funding: • 63 DEPSCoR awardees (12%) received their first NSF funding subsequent to their first DEPSCoR funding

Legislative Mandate for Study • Tangible results and progress toward the objectives of the program • Applications used by, or supportive of, operational users • Expand national research infrastructure • Activities consistent with statute • Activities were found to be consistent with statute • Assessment of program elements • Available data on DEPSCoR program activities and outcomes are insufficient for monitoring and evaluation purposes • Assessment of activities of state committees • State committees prioritized proposals that met state infrastructure development goals and reflected the mission/research needs of DOD • Committee processes varied widely from state to state and limited and variable data prevented detailed assessment • Advantages and disadvantages of institution & state based formulas • Mechanisms for improving the management and implementation of the program

Approach Taken • Assessment focused on comparing the current state-based formula with an institution-based criterion based on a maximum threshold for DOD S&E research funding • As directed in the study legislative mandate, particular emphasis was given to supporting defense missions and expanding the nation's defense research infrastructure • Since the legislative charge for the assessment does not specify how “expanding the nation’s defense research infrastructure” should be interpreted, the assessment considered advantages and disadvantages using a variety of possible interpretations • Assessment also considered alternative criteria: • Indicators of state-level S&T capacity (SEI, Milken Institute) • Normalization by state population (Census) • State-based and institution-based criteria can be combined • PI-level criteria are also possible

Effect of Institution Based Funding Threshold • 77 institutions in 2008 DEPSCoR-eligible states (360 total) received nonzero research funding from DOD in 2005 • 38 Carnegie “Very High” or “High” research institutions in DEPSCoR states • $5 million threshold would make 269 institutions eligible • Twelve of the 77 institutions in currently eligible jurisdictions (e.g., University of Delaware, University of Nevada, Brown, Clemson, Vanderbilt, University of Nebraska) would become ineligible • Considering only Carnegie “Very High” or “High” research universities, shift would increase number of eligible universities from 38 to 121

Definition of “Expanding National Research Infrastructure” • If interpreted as increasing equity in funding among states or achieving state-level infrastructure goals, a state-based formula would be advantageous • Current state-based formula for eligibility harnesses the state EPSCoR committees to coordinate infrastructure and capacity-building at the state level • If intent is to increase the competitiveness of historically-underrepresented states, eligibility can easily be determined at a state level • If interpreted as involving new investigators or institutions in defense-related research, an institution-based formula would be advantageous • Allows targeting of programmatic resources toward investigators at institutions that have not historically built relationships with DOD • Approach taken by late 1980s/early 1990s DOD Research Initiation Program • While a state-based approach includes the flexibility to channel DEPSCoR proposals toward historically underrepresented universities or new investigators within an eligible state, the institution-based approach allows greater flexibility to target underrepresented universities and investigators throughout the entire country • Could not be determined whether state-based or institution-based approach would elicit more qualified applications to support defense missions • Larger number of eligible institutions implies more proposals, but quality indeterminate

Effect of State Population Normalization ’05 DOD S&E R&D Funding, by State ’05 Per capita DOD S&E R&D Funding, by State • Green states were eligible in 2008, red states are “graduates”, blue states were never eligible • Comparing top chart with bottom shows dramatic difference in order • DEPSCoR graduates among highest per capita recipients • Several DEPSCoR-eligible states (e.g., AK, DE, ND, RI, MT, SD) above average in funding per capita

Other Capacity Indicators • Some DEPSCoR-eligible states (e.g., NH, RI, DE, KS) above average in Milken Institute state S&T index • Some “graduated” states below average • Some DEPSCoR-eligible states (e.g., DE, KS, NE, NH, RI, VT) in the top or second quartiles for more than half of Science and Engineering Indicators’ seven R&D output measures

IDA Recommendations • DOD should change the current process for review of proposals to focus more heavily on investigators’ future potential to conduct research rather than on their current research capabilities • DOD program managers should be formally encouraged to serve as mentors and facilitators for DEPSCoR investigators seeking to engage in further defense-related research • DOD should create data systems that will allow systematic tracking of DEPSCoR activities and outcomes • Congress should re-examine and consider clarifying ambiguities in the DEPSCoR legislative mandate • Once the DEPSCoR objectives have been clarified, redesign the program with a strategy for enhancing competitiveness at relevant level (e.g., individual, institution, state)

Some DEPSCOR Legislative Language Is Inconsistent • Objectives specify that the research institution is the level at which competitiveness is to be enhanced but authorizing legislation also specifies that eligibility for DEPSCoR be determined at the state level • 2008 change to eligibility criteria leaves ambiguous whether Congress intends DEPSCoR to target competitiveness at the institution or state level • Objectives specify that goal is to increase probability of long-term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal governmentbut eligibility determined based on DODfunding • Statute does not discuss whether Congress intends DEPSCoR should focus narrowly on [6.1] research that is relevant to DOD missions and priorities although this is how program is run

Recommendations for Legislative Clarification • To ensure that the program is implemented in a manner that meets current legislative priorities, Congress should clarify whether • The program is intended to increase competitiveness for federal research funding in general or for particular types of research funding • The program is intended to fund primarily basic research, primarily applied research, or a combination • The primary unit at which competitiveness should be enhanced is the institution, state, or other (e.g., individual investigator) • Eligibility criteria, funded activities, and other program elements should be structured in accordance with the program’s objectives

Competitiveness can be Enhanced at Different Levels • DEPSCoR supports individual or small-group research projects and can therefore be understood to primarily target capacity-building at the level of the individual • Though supporting training and purchase of equipment target capacity-building at level of the institution as well • DEPSCoR also operates at state level (e.g., involvement of EPSCoR committees, state-based eligibility criteria) • While it might be argued that institutional competitiveness depends on individual competitiveness and state competitiveness depends on institutional competitiveness, these dependencies are neither straightforward nor self-evident

Structure Program to Enhance at the Desired Organizational Level • Assessment identified other programs that might serve as potential models for DEPSCoR • State-level: NSF EPSCoR/Research Infrastructure Improvement (RII) Awards • One per state; integrated into state S&T plan; EPSCoR committee involvement; capacity building rather than research • Centers:NIH IDeA/Centers of Biomedical Research Excellence (COBRE) • Large-team research, equipment, mentoring; milestones for transition to support by standard NIH mechanisms • Investigator-level: DOD Research Initiation Program (RIP) or NIH/IDeA Academic Research Enhancement Award (AREA) Program • Institution-based eligibility criteria; small research awards to individual investigators; build relationships/expertise to allow for transition to support by standard research mechanisms

Backup Slides

State Eligibility over Time Source: IDA analysis of DEPSCoR BAAs and DOD DEPSCoR press releases Note: Red cells denote years in which jurisdictions were not eligible, and blue cells denote years in which states were eligible. Note: Missouri, which was eligible only in the 1993-4 competition, was not included in the table.

No Obvious Correlation Number of DEPSCoR awards and program funding Linear regression model of DEPSCoR state share of DOD university funding

DEPSCoR Emphasis is on Basic Research • Research oriented towards developing operationally useful devices or components is not a legislatively-mandated program goal • DEPSCoR projects mostly fund basic research • Not a statutory requirement • Administered by basic research organizations within Air Force and Army, plus Office of Naval Research (ONR) • Interviews with DEPSCoR program officers suggest that more applied research does not perform well in review • DEPSCoR program officers tend to track transitions to 6.2 or 6.3 research, but not further • Two transitions to operational use were found

Two Transitions to Operational Use Were Found • Ronald DeVore, University of South Carolina: Wavelet mathematics for image compression for tactical applications • DeVore and colleagues collaborate with program managers at Naval Air Warfare Center NAWC at China Lake to deliver wavelet-based image processing platform • Charles Creussere of NAWC implements wavelet-based image processing system for navigation in the Tomahawk Block II program • Michael Pursley, Clemson University: wireless, mobile, distributed, multimedia communication networks • Pursley and colleagues working since 1970s with ITT on tactical radio development • Group used DEPScoR funding to support research that provided better anti-jam communications and greater multiple-access capability • Research led to the Soldier Level Integrated Communications Environment (SLICE) wideband networking waveforms that have been integrated into the SINCGARS radio

Assessment of Defense Experimental Program: Results and Recommendations