Human Resources Outcome Indicators: Approaches and Prospects

1. Human Resources Outcome Indicators:Approaches and Prospects Colloquium on Measuring the Impacts of Science June 17-18, 2004 Montreal, Canada Rolf Lehming <rlehming@nsf.gov> Division of Science Resources Statistics, National Science Foundation NOTE: The views expressed are those of the author and do not represent the position of the National Science Foundation. No endorsement by the National Science Foundation should be inferred.

2. Two types of research output • New knowledge in the form of research results, methodological advances, conceptual breakthroughs • People who have gained substantive, conceptual, methodological and tacit knowledge along with a perspective on science and research • Both can theoretically be traced and analyzed • Bibliometrics as knowledge output proxy • Graduate enrollment, advanced degrees, and postdocs as proxies Human Resources Outcome Indicators

3. Longer-term research outcomes • Individuals’ research experiences and encounters with science may yield outcomes like • An appreciation of the scientific perspective • An openness to empirical inquiry • The acquisition of specific skills • The sum of these individual outcomes may yield • A technically sophisticated population and workforce • An educated citizenry able to deal with the science and technology components of public policy issues Human Resources Outcome Indicators

4. NSF Strategic Goals • “People” as one of NSF’s four Strategic Performance Goals • (along with “Ideas,” “Tools” and “Administration and Management”) • The strategic “People” outcome goal is to develop a workforce of scientists and engineers that is • Diverse, internationally competitive, and globally engaged • And to develop well-prepared [scientifically literate?] citizens • The plan includes annual performance goals and prospective and retrospective reporting foci Human Resources Outcome Indicators

5. NSF Administrative Data Number of people involved in NSF’s activities FY2002 FY2003 FY2004 _____________________Actual Estimated Estimated Senior researchers 28,960 29,820 30,590 Other professionals 12,060 12,180 12,640 Postdoctoral associates 5,740 6,060 6,170 Graduate students 26,170 27,440 28,690 Undergraduate students 34,250 32,710 36,350 K-12 students 11,460 13,640 14,640 K-12 teachers 84,710 85,460 86,830 Total number of people 203,350 207,310 215,910 Human Resources Outcome Indicators

6. NSF Administrative Data • These figures are compiled from proposal budgets estimating number and types of people involved • Limited personal information is captured about the principal investigator(s) only • NSF asks principal investigators to fill in, as part of project closeout, a matrix of project participants by type of position and demographic characteristics • Submission is voluntary and compliance variable • Figures do not include the almost 200,000 reviewers of the 30,000-plus proposals NSF receives annually Human Resources Outcome Indicators

7. NSF Administrative Data • The data provide approximate estimates of the number of people “reached,” in some fashion, by NSF funds or, in the case of peer reviewers, by NSF programs • These initial project estimates • are not exact • cannot be aggregated over time (the same persons may be counted in 2, 3, or more adjacent years) • in their current form invite no further analytical use Human Resources Outcome Indicators

8. NSF’s Strategic People Goal • NSF discussion about requiring investigators to supply demographic and contact information for project participants to allow use for • Tracking of careers • Studies of (scientific) productivity • Program assessments Human Resources Outcome Indicators

9. NSF’s Strategic People Goal • NSF declined for reasons including.(???) • Concern about PI response burden • Feasibility and legality (informed consent, university role) • Rules for establishing and maintaining Federal Systems of Records • Confidentiality and privacy concerns Human Resources Outcome Indicators

10. NSF’s Strategic People Goal • NSF emphasizes HR output and outcomes in proposal guidelines • “Proposals must describe … the broader impacts resulting from the proposed activities …” • Integration of research and education • Participation of underrepresented groups • Infrastructure enhancement such as facilities, instrumentation, networks, partnerships • Enhancement of scientific and technological understanding • Potential benefits of the activity to society at large Human Resources Outcome Indicators

11. NSF’s Strategic People Goal • NSF also emphasizes the importance of HR output and outcomes in reviewer instructions • Reviewers are instructed to address both merit review criteria • Intellectual merit of the proposed activity • Broader impacts [i.e., as described before] • Response to the broader-impacts criterion by proposal submitters and reviewers has been uneven • NSF has begun returning some proposals • NSF has repeatedly urged reviewers to address the second criterion Human Resources Outcome Indicators

12. NSF Surveys • Three biennial surveys of individuals who have at least a bachelor’s degree in science, engineering, or math or work in an S&E occupation • National Survey of College Graduates samples people with a bachelor’s or higher degree drawn from decennial U.S. Census, then followed up • National Survey of Recent College Graduates samples those with newly awarded S&E degrees • Survey of Doctorate Recipients draws a sample from a census of recipients of U.S. S&E doctorates • Integrated into the Scientists and Engineers Statistical Data System (SESTAT) Human Resources Outcome Indicators

13. NSF Surveys • Extensive demographic, educational, work-related, and personal information • Permits some outcome-related analyses including • Expansion of S&E occupations in the labor force (4 times faster growth than total LF over 2 decades) • Tracking of S&E degree holders into occupations not classified as science or engineering • High degree of relatedness of job, training • May indicate growing technical requirements of the economy • For individuals, education and career histories Human Resources Outcome Indicators

14. NSF Surveys • Outcome type variables but only inferred link to input, output items • Other limitations include • Expanded NAICS coding but aggregating up in analyses and barring of firm-level reporting • Limited ability to do productivity studies because of the paucity of output indicators • Matching with data from external sources faces serious difficulties • No ability to address citizenship competency issues • Separate survey on public attitudes Human Resources Outcome Indicators

15. Other NSF Survey • Survey of Industrial Research and Development • 25,000 firms, 2 personnel items • total payroll employment • Estimated FTE number of scientists and engineers working on R&D • Excludes social sciences and psychology • Analyses of the distribution of FTE researchers by firm sector and characteristics including R&D volume • In principle, trends in use of researchers across industries • No human resources detail available Human Resources Outcome Indicators

16. Non-NSF Surveys • The Bureau of Labor Statistics’ Occupational Employment Survey • Tracking of trends in S&E occupations at industry level in the framework of total employment • Potential to develop comparable state-level estimates (technical issues to be resolved) • Potential outcome indicators on changing size, composition, location of high-technology segments of the economy • BLS defines high-tech industries by S&E jobs percentage at least twice that of the average for all industries. Human Resources Outcome Indicators

17. Bottom Line and a Note • These surveys provide some means of estimating the size and some composition of the S&E workforce • All surveys but Industrial R&D Survey have only implied link of outcomes to inputs and outputs (except education inputs) • Limited ability to conduct in-depth panel analyses of career patterns in the context of the total labor market • Limited ability to conduct productivity studies  • A note on intergenerational social mobility • Valued mark of an open society • Education traditionally a strong factor • Role of science? Human Resources Outcome Indicators