Evaluating Nationally Funded Summer REUs

1. Evaluating Nationally Funded Summer REUs Avril Smart

2. Road Map • What is an REU? • Article 1: Structural Engineering REU • Article 2: Mental Health REU • Article 3: Evaluation of REUs • Article 4: Behavioral Sciences REU • Article 5: Chemical Engineering

3. What is an REU? • The Research Experiences for Undergraduates (REU) program supports active research participation by undergraduate students in any of the areas of research funded by the National Science Foundation (Sometimes NIMH). REU projects involve students in meaningful ways in ongoing research programs or in research projects specifically designed for the REU program. This solicitation features two mechanisms for support of student research: (1)REU Sites are based on independent proposals to initiate and conduct projects that engage a number of students in research. REU Sites may be based in a single discipline or academic department, or on interdisciplinary or multi-department research opportunities with a coherent intellectual theme. Proposals with an international dimension are welcome. A partnership with the Department of Defense supports REU Sites in DoD-relevant research areas. (2) REU Supplements may be requested for ongoing NSF-funded research projects or may be included as a component of proposals for new or renewal NSF grants or cooperative agreements.

4. Article 1 • For 7 years the University of Alabama has hosted a 9 week REU summer program for students in structural engineering • Objectives • Intro to research (research skills, confidence) • Oral, written and web presentations • Intro to Ethics • Collaborative learning environment • Publish work • Recruitment for graduate school • Norbert, D. (2004). Undergraduate Summer Research in Structural Engineering. Journal of Professional Issues in Engineering Education and Practice, p. 37-43.

5. Article 1: Evaluation Method/Design • Pre/Post and Follow up Survey

6. Article 1: Evaluation Results • Students were confident in abilities to complete undergrad degrees • Program successfully introduced student to research and ethics • Program did not entice uncertain students to pursue graduate school • Need to improve student placement • “Many” past participants have pursued graduate school • “Survey results indicate that the first five objectives have been met”

7. Article 2 • The Mental Health Institute (USF) developed a pilot tested, 6 week program for undergraduate research • Objectives • Strengthen research skills • Research skills/methods in behavioral health services • Learning experience for students and mentors • Pursue graduate school/ careers in behavioral science Gum, A., Mueller, K., Flink, D., Siraj, S., Batsche, C., Boothroyd, R. & Stiles, P. (2007). Evaluation of a Summer Research Institute on Behavioral Health for Undergraduate Students

8. Article 2: Evaluation Method/Design • Pre/Post Design and Follow up survey (T-Test) • Kirkpatrick’s educationally based evaluation process • Perceptions, learning, behavior and organizational Impact • 37 Close ended and open ended questions • Mentor Post test survey • 20 questions focusing on mentor interactions with participants/other faculty and future plans to participate in REU • Participant Narratives • Knowledge Tests • 75 tue false questions • 6 Month Survey

9. Article 2: Evaluation Results • Students increased knowledge in research skills • 62% pre to 70% post • Perceptions of Program • 75% thought this REU was better than other s they participated in • Mentorship • Over half rated mentorship experience as excellentand appreciated the mentor feedback • Overall mentors thought REU was either “excellent” or “good” • Future Plans • Follow-Up • 62% of participants reported either fully or partially applying skills learned in REU after participation

10. Article 3 • 3 year qualitative study which aimed to: • “clarify, and estimate the relative importance to students of the benefits of “good” undergraduate research experiences and the processes whereby these are achieved in a sample of science disciplines from the viewpoints of participating and non-participating undergraduates and faculty” • 76 students who had participated in one of the programs in 8 difference sciences at 4 different colleges which basically run a similar program • Goal is to clarify the what the programs entail and the benefits of them to students and faculty who both do and do not participate • Need to find direct benefits • What ‘good’ research experiences involve • Also want to look at minority representations in the students undertaking the programs

11. Article 3: Evaluation Method/Design • Qualitative method to research longitudinally 2 of 3 years • Focus groups and interviews • Sample size is 139 • 60 and 90 minutes and the third interview was done by phone since the students had graduated from their undergrad degree

12. Article 3: Evaluation Results • Findings to confirm many of the benefits that the programs claims to provide but in other ways do not • Some benefits are only relevant to small numbers of the students • For example of a few students went to conferences and presented • Gains in communication were the greatest • Encouragement for direction of careers or graduate school was not supported • But did clarify what they wanted to do • Many of the students describe specific benefits • The research is a educational and personal growth experience with many benefits • Students deepened their understanding and learnt to apply their knowledge more • Gains in confidence have them empowerment to make a difference in their field

13. Article 4 • American Indians Into Psychology program (AIIP) • 8 week, 40 hour per week NSF funded Summer REU • Objectives • Overview of the research process • Formulating hypotheses, integrating/analyzing literature, designing a study, collecting/analyzing data, presenting findings Page, M. & Abramson, C. (2004). The National Science Foundation Research Experiences for Undergraduates Program: Experiences and Recommendations. Teaching of Psychology, p.241-246.

14. Article 4: Evaluation Method/Design • Pre/Post Survey Evaluation (n=24 pre and n=21 post) • Undefined survey measure on “usefulness of program information” • Method of Analysis • Descriptive Statistics

15. Article 4: Evaluation Results • All students interested in graduate school pre program (33% clinical psych pre , 23% clinical psych post) • Various student accolades • Publications • Conference presentations • Usefulness • Grant Writing Session • Presentation workshops • Team Building • GRE Prep

16. Article 5 • Sustainability Energy and Engineering (SEE) REU • 14 students, 8 week program • Objectives • 1) develop future knowledge leaders in sustainability and green engineering • 2) enhance the likelihood of successful graduate education in engineering among students of color and • 3) promote disciplined thinking related to science among students.

17. Article 5: Evaluation Method/Design • Evaluation Purpose • (1) skills and knowledge • (2) attitudes towards research • (3) effective mentoring • (4) programmatic issues • Design • Mixed Method (Qualitative/ Quantitative) • Pre/Post (Follow-up) • Focus Group (3)

18. Article 5: Evaluation Results Results • Moderate success of minority group recruitment • Students were high achieving and showed promise for graduate training • Had clearly defined interest in sustainability research • Ethics seminars showed great improvement • Lower representation of minority groups compared to previous years • On a whole mentor/student relationships were positive but there were still some problems • Student’s expectations were generally met

19. Summary • There is much variability between the programs • Not much agreement exists on how to structure or evaluate the programs although most believe they are effective • Mismatch between the objective and the evaluation outcomes • Purpose of the evaluation is rarely disclosed • Measurement tools are not necessarily sound