Presentation made to SoTE Conference March 28, 2014

1. Using Assessment of Undergraduate Research to Improve Programming and Student Outcomes Presentation made to SoTE ConferenceMarch 28, 2014 Susan Larson

2. What is undergraduate research, scholarship & creative activity (UR)? • Council on Undergraduate Research: An inquiry or investigation conducted by an undergraduate student that makes an original intellectual or creative contribution to the discipline. www.cur.org • Concordia College: Undergraduate research, encompassing all forms of scholarship and creative activity, is an investigation or inquiry conducted by a student, under the mentorship of a faculty member, which contributes to a high-level intellectual or creative outcome.

3. Defining UR • Student, process centered  Outcome, product centered • Student initiated  Faculty initiated • All students Honors students • Curriculum based  Co-curricular fellowships • Collaborative  Individual • Original to the student  Original to the discipline • Multi- or interdisciplinary  Disciplinary based • Campus community audience  Professional audience - Beckman & Hensel, CUR Quarterly, 2009

4. Value of UR • Cognitive and intellectual growth • Gains in knowledge and skills • Academic achievement and educational attainment • Professional growth and advancement • Personal growth and development

5. Assessing the Benefits of UR • Survey of Undergraduate Research Experiences (SURE) • Includes demographic variables, learning gains, and evaluation of aspects of summer programs. • Students report the greatest learning gains on items related to the research process, scientific problems, and lab techniques • Students report learning gains on personal development measures (e.g., tolerance for obstacles and working independently) • Summer research sustains an interest in pursuing science • Lopatto (2004), Cell BiolEduc, 270-277. • http://www.grinnell.edu/users/lopatto

6. Assessing the Benefits of UR • Assessment of undergraduate research opportunity program (UROP) at Univ. of Michigan • Longitudinal assessment evaluating student retention, academic performance, engagement, pursuit of graduate and professional education • Evaluated UROP students and matched control group: individuals who applied, but did not participate • Significant positive impact of UROP on degree completion rates (most notably for African American students) and likelihood of pursuing post-graduate education • Nagda, Gregerman, et al. (1998), The Review of Higher Education, 55-72 • http://www.lsa.umich.edu/urop/aboutus/evaluationassessment

7. Assessing the Benefits of UR • Ethnographic study of summer undergraduate research (UR) experiences • Initial work done at four liberal arts colleges • Focus groups with students and faculty • Both groups described gains related to process of "becoming a scientist" • Faculty focused on “professional socialization” • Students more focused on “personal and intellectual development” • Hunter, Laursen, Seymour (2007). Science Education, 36-74. • http://www.colorado.edu/eer/index.html

8. Why Should YOU Assess UR? • Evaluate the effectiveness of your program • Determine if your programming is meeting your objectives • Develop your programming • For donors, your administrators, grant writing • Understand the benefits of UR for your students • To improve student learning in your context • To improve faculty mentoring • To broaden participation in UR • But, we don’t have to re-invent the wheel

9. What Should You Assess? • Student (learning) outcomes • What did you learn in this experience? • How did this affect your educational experience, career, etc.? • Strategies for developing an assessment program • Example of assessment program at my home institution • Program effectiveness • Make use of COEUR • Characteristics of Excellence in Undergraduate Research

10. Questions to Ask • What do you want to know? • Should align with the objectives of your program • Student researchers more likely to go to grad school? • Self-confidence and self-efficacy? • Impact on departments/advisors scholarly activity? • Who is the target of your assessment? • Who does the assessing? • What is the quality of the instrument? • How will you use the results? • Benchmarking • Program improvement • Understanding student learning outcomes

11. Assessment of UR and SoTL • There are great opportunities for SoTL projects related to undergraduate research • So, consider framing your UR assessment to answer the questions you need to ask, but also the questions you want to ask. • E.g., benefits to faculty: often discussed, less often assessed • E.g., how is UR different/similar to other forms of engaged learning at UVU?

12. Designing a Sustainable Assessment Program • State expected outcomes, clear and measurable • Research skills, student development goals • Program goals • Determine methods and criteria to assess outcomes • Direct assessment and/or indirect assessment • Qualitative and/or quantitative assessment • Start small, start with successes, minimize the burden • Identify and collect information • Set priorities, conduct periodic and timely assessments • Use the data! • Share results, use assessment results in planning and progam evaluation

13. The Assessment Cycle • Assessment of UR is an on-going process of • Establishing clear, measurable expected outcomesfor UR programs • Providing participants (students, faculty, institutions, system) with opportunitiesto achieve those outcomes • Systematically gathering, analyzing, and interpreting evidenceto determine how well UR activities match our expectations • Using the resulting information to improveUR programs and initiatives • Maki, Journal of Academic Leadership, 2002

14. Example of an Assessment Plan for UR: Concordia College • The assessment plan for the Office of Undergraduate Research seeks to accomplish the following goals • Provide a count of undergraduate research, scholarship and creative activities at Concordia, including demographic data • Assess learning achieved in undergraduate research experiences • Evaluate the effectiveness of the Office of Undergraduate Research.

15. Counting UR • Faculty surveys administered by the Office of Institutional Research • Faculty report on • # of students they mentored • # of projects they mentored • Funding for the projects • Outcomes of the projects, e.g., presentations, conferences • Survey also distinguishes classroom-based and individually-mentored research

16. Other Ways to Count UR • Enrollment in directed research courses • # of students participating in funded travel opportunities • # of students presenting at on-campus symposia or related events • Course syllabi which note an extensive research experience • Count of students who engage in off campus research • Make use of department annual reports • OTHERS?

17. The Challenge and Value of the Count • Do you need to count everyone? • Can we even do it if we tried? • Is counting enough? • Do you need to know what happens to these students? • Do you need to know their demographic characteristics?

18. Assessment of Student Learning • Students are expected to achieve some of the following learning goals/objectives. • formulate questions in their field • demonstrate ability to understand literature • explain methodology • demonstrate proficiency in the use of the tools and instruments of the area of study • analyze and interpret literature and/or results as appropriate to their field of study • demonstrate ability to communicate (writing, presenting) within the area of study, and

19. Assessment of Summer Research Program • Indirect assessment -- student surveys • In-house self-report assessment of learning gains and confidence • Responses to open-ended question about learning outcomes • Evaluation of the degree to which students met the goals in the learning agreement they sign with their mentor at the start of the summer. • Science students complete Lapatto’s SURE survey (http://www.grinnell.edu/academic/csla/assessment/sure).

20. Assessment of Summer Research Program • Faculty mentor survey • Faculty rate their student’s learning gains and respond to open-ended questions about learning gains • Faculty and students rate their satisfaction with the summer program activities.

21. How Much Did You Gain as a Result of Your Summer Research Experience?

22. Meeting Learning Agreement Goals

23. Open Ended Question • “What were one or two of the most important things you learned from your research experience?” • Research skills specific to specific projects • Understanding of the research process • E.gs., interpreting data; reading scientific literature; the importance of a detailed lab notebook. • Understanding of science • E.g., understanding how scientists work together; the importance of previously published findings; appreciation of the differing perspectives individuals from different disciplines bring to a problem • Personal gains • Patience; honesty; flexibility; independence; the importance of team work.

24. Summer Program Activities • Allowed them to develop some practical skills • Appreciated the chance to interact with other student researchers • Some felt that the workshops targeted first-time researchers and took too much time away from the lab

25. Other Ways to Assess SLOs • Institutional assessment data • E.g., NSSE and critical thinking measures employed by the institution • Departmental assessment programs • Nationally available survey’s • E.g., SURE and others by Lapotto • E.g., Undergraduate Research Student Self-Assessment (URSSA), on salgsite.org • Rubrics for scoring posters/talks given by research students • Reflective essays for students who travel and present UR • Focus groups

26. Other Ways to Assess Student Outcomes • Post graduate activities • Alumni survey • Career center survey • Publications and presentations generated by involvement in UR • Review CUR’s assessment database for ideas • https://www.zotero.org/groups/curassessmentbibliography/items/

27. Using the Assessment Data • Closing the assessment loop • Lower scores on communicating research • Added an oral presentation expectation to the summer program • Lower rating on learning gains related to ethics • Will enhance summer program workshop on ethics • Information about mentor availability and its impact on the summer experience • Conversations with mentors about their expectations

28. Program Evaluation • May want to go beyond assessing the experience of students to understand if the UR program you offer is robust and effective • Could be accomplished with the assistance of COEUR http://www.cur.org/publications/publication_listings/coeur/

29. What is COEUR? • A summary of best practices that support and sustain highly effective undergraduate research environments • The purpose of COEUR is to provide a guide for those who wish to build, evaluate, and maintain robust, productive, meaningful and sustainable undergraduate research programs • Function as a guide for strategic planning • Provide programmatic and institutional benchmarking against national standards • Outline a structure for UR self-study • Provide a framework for assessment and evaluation of UR

30. The Characteristics • Campus mission and culture • e.g., institutional commitment • Administrative support • e.g., UR program office • Research infrastructure • e.g., space, instrumentation and equipment • Professional development opportunities • e.g., research leaves, mentorship training

31. The Characteristics • Recognition • e.g., UR in promotion and tenure guidelines • External funding • e.g., internal and external funding for research • Dissemination • e.g., peer-reviewed publication, exhibition, or performance; student research conferences • Student-centered issues • e.g., community of student scholars

32. The Characteristics • Curriculum • e.g., integration of teaching and research; student course credit for research • Summer research program • e.g., faculty & student compensation • Assessment activities • e.g., assessment of student learning • Strategic Planning

33. Simple Evaluation Rubrics http://www.cur.org/publications/publication_listings/coeur/

34. Conclusions • There are many published benefits of UR • But, local, contextualized assessment of UR are important • When undertaking an assessment program • Know what you are assessing and why • Keep it simple and sustainable • Don’t assess everything all the time • Keep your focus on enhancing student learning • Use assessment data and program evaluation to improve your UR programming • COEURcan be a useful tool for improving the experience you offer students