CTSA Evaluation Steering Committee An Overview

1. CTSA Evaluation Steering Committee An Overview Harold Pincus, Columbia Chair, CTSA Evaluation Steering Committee Lori Mulligan, NIH Coordinator CTSA Evaluation Steering Committee

2. Purpose • Coordinates institutional evaluation efforts with the national evaluation of the CTSA program. • Provides a forum to exchange information about institutional evaluation approaches, challenges, and progress.

3. Evaluation Steering Committee Composition CTSA Representatives & PI Liaisons National CTSA Evaluation Steering Committee NIH CTSA Evaluation Subcommittee Independent Evaluators

4. Evaluation SteeringCommittee& Workgroups

5. Institutional Evaluation Workgroups • Definitions and Measurement (Chair: B. Trochim, Cornell) • Identify, prioritize, analyze potential definitional constructs (i.e., translational, disciplinarity, collaboration, etc) • Synthesize definitions and measurement approaches • Prepare evaluation briefs/whitepapers that summarize sound measurement approaches, tools, and methods • Work with other workgroups, committees as appropriate

6. Institutional Evaluation Workgroup • Social Network Analysis (Chair: J. Rainwater, UC Davis) • Explores SNA as a tool to assess communication and collaboration • Visually depict perceived strength of relationships among individual groups, identifying strong and weak connections among them • Show changes over time in relationships, which in turn can demonstrate how the CTSA program is affecting the way individuals or groups are working with each other. Two SNA Examples: • Relationships among key functions - S. Lurie (Rochester) • Co-authorship /betweenness among obesity investigators - S. Johnson (Columbia) *Administrative Supplement: “Using Social Network Analysis to Evaluate Translational Collaboration” to synergize with Communications SC proposal: “Exploring Web-Based Methods to Maximize Sharing in the CTSA Consortium” 

7. Institutional Evaluation Workgroups • Shared Resources (Chair: J. Segal, Johns Hopkins) • Identify and share evaluation resources (i.e. surveys, needs assessments, user satisfaction, culture change) • Create a community of committed individuals willing to share instruments as well as processes • Examples: https://www.ctsawiki.org/wiki/display/Eval/Shared+Resources+Workgroup • IRB Issues and Evaluation (Chair: P. Moberg, U of Wisconsin) • Workgroup formed as an outcome from Feb 8 face-to-face meeting (see next slide)

8. Highlights from Feb 8 Face-to-Face MtgRound Robin Sessions • IRB Issues and Evaluation Workgroup Chair: P. Moberg, U Wisconsin • Consider possibilities to harmonize across institutions • Explore discussions with other CTSA Steering Committees (e.g., Alternate IRB models, Regulatory, Clinical Research Ethics) • Identify ways to achieve greater transparency through: • Set of tools to educate IRBs about evaluation • Example of Key Features that an IRB would review for exemption • Convene a subset of CTSA evaluators to engage with American Evaluation Association (Scheduled for November 7 at AEA Conference)

9. Highlights from Feb 8 Face – to – Face MtgRound Robin Sessions • The Role of the Evaluator (Facilitator: D. Rubio, U Pittsburgh) • Discussed different evaluation approaches underway – core performance v. program evaluation; external v. internal consultants • Identified challenges – how best to capture real change/transformation; how do you evaluate leadership and to whom and how are results reported • Consider what role the evaluator has or should have with external advisory board

10. Highlights from Feb 8 Face – to – Face MtgRound Robin Sessions • Evaluating Indirect Benefits of the CTSA (Facilitator: A. Dozier, Rochester) • Consider economic impact • Ability to draw different skill sets into C & T research • Assess nature of new partnerships and ability to attract unexpected collaborators • New ways to engage communities • Assess cultural changes • Generate examples of indirect effects and post on Wiki (i.e., Rochester economic impact)

11. Schedule of Upcoming Meetings Evaluation Steering Committee • August 20, 2008, 2:30 – 4:00 ET • October 15, 2008, 2:30 – 4:00 ET • November 8 – 9, 2008, Denver, Face-to-Face meeting • Held in conjunction with the American Evaluation Association Annual Conference that will feature 18 CTSA presentations, 22 CTSA evaluators, and 10 CTSA institutions Standardizing and Operationalizing Definitions • September 9, 2008, 4:30 – 5:30 ET • October 7, 2008, 4:30 – 5:30 ET • October 21, 2008, 4:30 – 5:30 ET Shared Resources • September 11, 2008, 3:00 – 4:00 ET • October 9, 2008, 3:00 – 4:00 ET • November 13, 2008, 3:00 – 4:00 ET Social Network Analysis • September 11, 2008, 3:00 – 4:00 ET • November 20, 2008, 3:00 – 4:00 ET

12. November 8-9, 2008 Face-to-Face Meeting Note: Please submit agenda topics and suggestions for breakout sessions on the Wiki: https://www.ctsawiki.org/wiki/display/Eval/Add+Agenda+Topics+-+Face-to-Face+Meeting+in+Denver-+November+8-9%2C+2008 Meeting Schedule (All times are local Denver time, i.e. MT) DateTime Saturday, Nov 8 6:30pm – 8:30pm Sunday, Nov 9 8:00am – 3:00pm Registration Registration will begin at 5:45pm on Saturday, November 8, with light refreshments beginning at 6:00pm. Registration will continue into Sunday, November 9 and will begin at 7:15am. Meeting Location Hilton Garden Inn Denver Downtown 1400 Welton Street Denver, CO 80202 Phone: 303-603-8000 Lodging (0.2 miles from the meeting location) Should you wish to reserve a sleeping room, please feel free to contact the Hyatt Regency where there is currently a room block for the AEA conference. The number for the Hyatt is 1-888-421-1442. Reservations must be made by October 6, 2008 but preferably earlier. Please mention the group name “American Evaluation Association” Room Block when making your reservation. You can also log on to http://www.eval.org/eval2008/08hotels2.htm and book your reservation online.

13. Next Steps for New Members • Join workgroups, recommend forming new ones, propose topics for webconferences and meetings • Participate in August 20 Full Steering Committee Webconference • Attend and help shape agenda forthe November 8 - 9 Evaluation Steering Committee meeting in Denver, CO • Held in conjunction with the American Evaluation Association Annual Conference that will feature 18 CTSA presentations, 22 CTSA evaluators, and 10 CTSA institutions

14. Evaluation SC Points of Contact Steering Committee Chair & Coordinators: • Dr. Harold Pincus, CTSA Chair E-mail: pincush@pi.cpmc.columbia.edu • Ms. Lori Mulligan, NIH Coordinator E-mail: mulligal@mail.nih.gov • Ms. Huda Aden, Project Manager E-mail: aden_huda@bah.com PI Liaisons to Evaluation SC: • Dr. Garret FitzGerald, University of Pennsylvania E-mail: garret@spirit.gcrc.upenn.edu • Dr. Robert Rizza, Mayo Clinic College of Medicine, PI Liaison E-mail: rizza.robert@mayo.edu

15. Evaluation SC Points of Contact Workgroup Chairs: • Dr. William Trochim, Chair, Definitions and Measurement Workgroup E-mail: wmt1@cornell.edu • Dr. Julie Rainwater, Chair, Social Network Analysis Workgroup E-mail: jarainwater@ucdavis.edu • Dr. Jodi Segal, Chair Shared Resources Workgroup E-mail: jsegal@jhmi.edu • Dr. Paul Moberg, Chair IRB Workgroup E-mail: btai@nida.nih.gov

16. Evaluation SC Resources CTSAweb.org: • http://www.ctsaweb.org/index.cfm?fuseaction=committee.viewCommittee&com_ID=19 CTSA Wiki: • https://www.ctsawiki.org/wiki/display/Eval/Evaluation-+Home