Bridges to the Future Technical Assistance Workshop

1. NIGMS Bridges to the Future Technical Assistance Workshop Division of Minority Opportunities in Research National Institute of General Medical Sciences National Institutes of Health, USDHHS June 19, 2009

2. Bridges To the Future Program Clifton Poodry, Ph.D. Director, MORE Division National Institute of General Medical Sciences National Institutes of Health, USDHHS June, 19 2009

3. Outline of Presentation Overview of Bridges to Baccalaureate (PAR-07-411) & Bridges to Doctorate (PAR-07-410) programs Guidance forBridges Program Development

4. Bridges Program Goals An institutional program with a focus on increasing the number of community college/master’s degree students from underrepresented groups and/or health disparities populations (URMs) who transfer and complete the baccalaureate/PhD degree, respectively, in biomedical and behavioral sciences.

5. The Bridges Program Emphasizes: Institutional Focus:impact on Bridges and non-Bridges students alike so more URMs transfer and complete the baccalaureate/PhD degree Partnerships: 2-year community college(s) with four-year institution(s), & master’s degree-granting institution(s) with PhD degree-granting institution(s) Developmental Activities: well-integratedactivitiesthat will provide students with the necessaryacademic preparation and skills to enable their transition andsuccessful completion of the baccalaureate/PhD degree inbiomedical/behavioral sciences Clear Expectations

6. Partnership/Consortium • Purpose:To facilitate a seamless transition of targeted students from associate to baccalaureate degree-granting institution, and from master’s to the PhD degree-granting institution in biomedical/behavioral sciences • Size:Up to 4 institutions for BTB, and 3 for BTD, including the applicant institution, unless strongly justified otherwise • An institution may participate in more than one Bridges partnership if strongly justified by the potential to magnify the programs’ and institution’s outcomes

7. Bridges Students (Students who receive support in the form of salaries/wages) • Students from groups underrepresented in the biomedical and behavioral research enterprise of the nation and/or populations disproportionately affected by health disparities (targeted groups). Nationally, these groups include, but are not limited to, African Americans, Hispanic Americans, Native Americans (including Alaska Natives), Natives of the U.S. Pacific Islands, and/or rural Appalachians. • Must be U.S. citizens or non-citizen nationals or permanent residents. • Must be matriculated full-time in associate/master’s degree program in biomedically relevant science fields at the partner community college/master’s degree institution.

8. Student Selection & Critical Mass • Student Selection • Institution’s responsibility to establish student qualifications • Describe the criteria for selection and retention of Bridges students into the program • Critical Mass: Number of students in the program each year • BTB: 15-20, with 4 from each CC • BTD: 6-8, with 3 from each master’s degree institution

9. Key Program Expectations • Baseline Data & Goals to Improve • Effective Partnerships • Strong Pool of Targeted Students • Well-integrated Student Development Activities year-round • Strong Institutional Commitment • Sound Evaluation Plan

10. What Are the Baseline Data & Measurable Objectives? Baseline • The baseline is a starting point; it serves as a guide to capacity, and is necessary to gauge the impact of the program. For example, an increase of 100% on a base of 1 is not nearly as impressive as if it were on a base of 10. • Usually it’s an average data over a 3-5 year period Measurable Objectives • These are brief (and focused) statements of end results – connected to the long-term goal. They can be qualitative and quantitative, but must be measurable. • They are not a means to an end, or a checklist of “to do” list. • The activities proposed are the means to achieve your goals and objectives.

11. Examples of Goals, Baseline Data & Measurable Objectives Goal:To increase the number of CC/master’s degree students who transfer to four-year/PhD degree-granting institutions in biomedically relevant sciences. Baseline:How many and what percent of the students currently transfer to baccalaureate/PhD degree programs? Measurable objective: What is the proposed number and percentage of students who will transfer to baccalaureate/PhD degree programs during the grant period?

12. Examples of Baseline Data & Objectives, cont. Institutional Baseline: Out of a total of 40 students in biomedically relevant fields (40% URM & 60% non-URM), 4 URMs (25%) & 9 non-URMs (37.5%) actually transfer to programs in biomedically relevant disciplines per year (average data from 2005-2008). Institutional Objective: The URM transfer in biomedically relevant disciplines will increase from the current rate of 25% to 37.5% (from an average of 4 to 6 students per year) by the fourth year of the grant award.

13. Examples of Baseline Data & Objectives, cont. Bridges Baseline on Transfer -An average of 4 out of 8 (50%) Bridges students transferred to programs in biomedically relevant disciplines per year (average data from 2005-2008). Objective -The transfer of Bridges students to programs in biomedically relevant disciplines will increase from the current rate of 50% to 75% (from an average of 4 to 6 students per year) by the fourth year of the grant award. Bridges Baseline on Degree Completion -An average of 2 out of 4 (50%) transferring Bridges students completed the baccalaureate/PhD degree in biomedically relevant disciplines per year (average data from 2005-2008). Objective -Bridges students’ degree completion in biomedically relevant disciplines will increase from the current rate of 50% to 75% (to an average of 4.5 graduates per year) by the fourth year of the grant award.

14. Goals to Improve the Track Record • Clearly stated goals, and measurable objectives relative to the baseline and in context of NIH expectations • Must improve on the previous record and make substantial progress towards meeting the NIH expectations during the next 5 years

15. NIH Expectations • Institutional • Increase in the institutional transfer of targeted students (Bridges & non-Bridges) to baccalaureate/PhD degree programs in biomedically relevant sciences by 50% in five years • Bridges • Increase in academic preparation and skills development • Increase in the transfer of students to baccalaureate/PhD degree programs (70-75% in five years) • Increase in the baccalaureate/PhD degree completion of transferring students (75-80% in five years)

16. BASELINE DATA AND OBJECTIVES

17. Examples of Developmental Activities To Move the Institution Forward from Baseline to NIH Expectations Bridges to Baccalaureate: • Developing community college courses and curricula that are fully transferable to the baccalaureate institution • Faculty from the four-year institution serving as visiting lecturers at the two-year institution, or developing joint team-taught courses • Introduction of research concepts into the community college curriculum and/or offering a research skills course to stimulate students’ interest in science • Supplementary instruction in “gate-keeping” courses at the community college • Mentored research experiences for the Bridges students, including summer research internships

18. Examples of Developmental Activities, Cont. Bridges to Baccalaureate, cont. • Skills development (e.g., critical thinking, communications skills, study and time management skills) workshops, and research careers seminars • Preparing community college students, through college orientation classes, etc., for transfer to the 4-year institution • Peer mentoring and tutoring, and research career seminars • Research education conferences for Bridges faculty (CC), and advanced or special coursesat the partner 4-year institution • Research conferences for 4-year faculty mentors, if accompanying students making presentations • Others

19. Examples of Developmental Activities Bridges to Doctorate: • Faculty from the two types of institutions jointly developing courses and curricula, including updating existing or developing new/advanced courses at the master’s degree institution • Faculty from the doctorate institution serving as visiting lecturers, offering lectures and/or laboratory courses at the master’s degree institution • Faculty from the doctorate institution providing mentored research experiences to Bridges students, including the summer research internships, and serving on their thesis advisory committees • Fostering research capacity of the master’s degree institution via research collaborations

20. Examples of Developmental Activities Bridges to Doctorate, cont. • Allowing Bridges students to take some courses, and complete part of their master’s thesis research at the doctoral institution • Providing Bridges students access to computer and library facilities, seminars, and workshops,etc., at the doctoral institution • Establishing a mentoring and academic counseling program for master’s students with faculty at the doctoral institution • Advanced or special courses and scientific research conferences for Bridges faculty from the master’s degree institution

21. Institutional Commitment Strong commitment to the goals of the proposed program from all participating institutions, including commitment to: • Provide the institutional data on the transfer of URMs and non-URMs in biomedically relevant sciences • Provide the data on transfer and baccalaureate/PhD degree completion by Bridges students in biomedically relevant sciences • Track Bridges students over a ten-year period as they progress through the pipeline • Institutionalize the most effective activities supported by the Bridges program

22. Evaluation • Purpose:Provide information useful to the applicant and the partner institutions for improving the program, and for institutionalizing the most effective activities supported by the Bridges program. • Provide a sound evaluation plan, with timeline, that’s in-line with measurable goals and objectives • Identify a qualified evaluator and include his/her biosketch

23. Consortium Agreement • Consortium agreement between the associate and baccalaureate, and between master’s and PhD degree-granting institutions define theparticipating institutions’respective rolesin administering the Bridges program. • The application must include a letter from each collaborating institution signed by the appropriate institutional officials and program director/program coordinator, acknowledgingparticipation in the program. These letters must also include the following: “THE APPROPRIATE PROGRAMMATIC AND ADMINISTRATIVE PERSONNEL OF EACH INSTITUTION INVOLVED IN THIS GRANT APPLICATION ARE AWARE OF THE NIH CONSORTIUM GRANT POLICY AND ARE PREPARED TO ESTABLISH THE NECESSARY INTER-INSTITUTIONAL AGREEMENT(S) CONSISTENT WITH THAT POLICY.”

24. Budget and Years of Support • The program provides support for student, faculty, and institutional development activities • Average awards range from $150,000 to $300,000 (DC) per year. • Budget must be reasonable, well documented, and fully justified and commensurate with the scope of the proposed program • Awards for up to five years

25. Guidance forBridges Program Development

26. Develop a Plan • Read the FOA and F&Qs • Conduct an institutional assessment and gather baseline data: how many transferring and graduating; why not more transferring and graduating? what can change so more will transfer and graduate? • What are the institutional needs? • What is your long range goal? • What are your measurable objectives? • What activities will help your institution achieve these objectives • How will you evaluate your program outcomes? • Follow all of the instructions in writing your application!!

27. Application Components/Organization • Project Summary, & Performance site (s) • Facilities & other Resources • Key Persons and their Biographical Sketches • Budget • Research Plan:

28. Application Components/Org., Cont. Research Plan: • (Introduction) • Specific Aims and Measurable Objectives • Background and Significance: Institutional & student data; vision and anticipated value • Preliminary Studies or Progress Report: Specific Outcome Data & Impact on the Institution • Research Design & Methods:

29. Application Components/Org., Cont. Research Design & Methods: • PD, PCs, and Program Faculty • Developmental Activities: • The rationale and detailed description • Who will implement and the timeline • Possible pitfalls and alternative approaches • Responsible Conduct of Research • Evaluation Plan • Consortium Agreements

30. Presentation of Data • Present data in figures, graphs, tables or text • Place figures, tables, and graphs close to where they are referred to in the text • Make all figures, tables, and graphs clearly legible • Avoid irrelevant information • Do not use the Appendix to circumvent the page limitations of the Research Plan

31. Common Reasons for Failure • Missing or inadequate baseline data • Lack of clear and well-defined measurable objectives • Lack of adequate progress • Activities poorly related to the objectives • Poorly developed or missing evaluation plan • Lack of institutional commitment or support • Lack of coordination with other institutional programs aimed at accomplishing similar goals • “Program-centric” application vs. institutional application

32. Summary: Bridges “Essentials” • Institutional and Bridges baseline data • Clear statement of program goals, specific aims, and measurable objectives • Detailed progress report, if applicable • PD/PI, Coordinators, and program faculty with appropriate training and experience • Detailed description of developmental activities • Sound evaluation plan (see for example, http://oerl.sri.com/ and http://www.nsf.gov/pubs/2002/nsf02057/start.htm) • A plan for training in responsible conduct of research • Evidence of strong institutional commitment • Consortium Agreement with “specific language” • A well justified budget that is commensurate with the scope of the proposed program

33. NIGMS Program Evaluation • BRIDGES TO THE DOCTORATE (PAR-07-410) • BRIDGES TO THE BACCALAUREATE (PAR-07-411) Clifton Poodry, Ph.D. Division of Minority Opportunities in Research National Institutes of General Medical Sciences National Institutes of Health, USDHHS June 19, 2009

34. NIGMS Evaluation: What is it? Program evaluations are individual, systematic studies that use objective measurement and analysis to answer specific questions about how well a program is working. - #GAO/GGD-00-204 Program Evaluation Program evaluation and the tracking of students are not the same thing.

35. NIGMS Program Evaluation Answers Questions Like…. • Does it work? • How well does it work? • Does it do what we want it to? • Does it work for the reasons we think it does? • Is it cost effective? • Are the benefits worth it? • What are the unintended consequences?

36. NIGMS Why bother? • Supports continuous program improvement • Increases understanding of the program – how are activities and strategies linked to results? • Leads to improved planning and management • Provides shared understanding of program

37. NIGMS Guidelines for Conducting Successful Evaluations • Invest heavily in planning early on • Use knowledgeable, experienced evaluators (usually social scientists) • Integrate evaluation into ongoing activities of the program

38. NIGMS Typical Evaluations Needs Assessment • What is nature & extent of the issues program should address? • Planning phase Process Evaluation • Is program being conducted & producing output as planned? • How can process can be improved? Outcome Evaluation • Extent to which a program’s goals have been met?

39. NIGMS Sample Study Questions

40. NIGMS Sample Study Questions

41. NIGMS Why should you care? • If you don’t know where you’re going, any road will take you there. • - Lewis Carroll, Alice in Wonderland -Illustration by Sir John Tenniel, eBooks@ Adelaide, 2004

42. NIGMS Key Steps in Evaluation • Engage stakeholders • Describe the program • Focus the evaluation design • Gather credible evidence • Justify conclusions (present data, analysis used, and findings) • Ensure use and share lessons

43. NIGMS 1. Engage Stakeholders Who are the stakeholders? Those involved in program operations, those affected by the program operations, and primary users of evaluation results

44. NIGMS • Describe the program • What are the goals and specific aims of the program? • What problem or need is it designed to address? • What are the measurable objectives? • What are the strategies to achieve the objectives? • What are the expected effects? • What are the resources and activities? • How is the program supposed to work?

45. NIGMS Remember: • Activity ≠ Program • You cannot evaluate a program by assessing only an activity

46. NIGMS Model of a Training Program Resources Activities Impact (Inputs) (Outputs) (Outcomes) What are the changes or benefits? What is done? What is invested? What is invested? Workshops & Seminars Short term Knowledge Skills Attitudes Intermediate Behaviors Practices Long term Enter PhD Program Faculty & Staff Money Training in scientific methods Equipment & Technology Mentoring by faculty member Research base

47. NIGMS 3. Focus the evaluation design • What do you want to know? (key questions) • Who will be involved in or affected by the evaluation or use the findings? (stakeholders) • To focus an evaluation, consider its purpose, uses, questions, methods, roles, budgets, deliverables etc. • An evaluation cannot answer all questions for all stakeholders

48. NIGMS 4. Gather credible evidence • Evidence must be • believable, trustworthy, and relevant • Select methodological approach & data collection instruments • Determine who is studied and when

49. NIGMS 5. “Justify” conclusions • Consider data: • Analysis and synthesis • - determine findings • Interpretation • - what do findings mean? • Judgments • - what is the value of findings based on accepted standards? • Recommendations – • - what claims can be made? • - what are the limitations of your design?

50. NIGMS An evaluation plan should include: • Program description with baseline data • Purpose & rationale for evaluation • Evaluation Design • Data Collection & Analyses • Products of evaluation & their use • Project Management • Budget estimate