1 / 20

2006 ANNUAL MEETING Arlington, Virginia

Evaluators' Breakout II: EVALUATING THE EVALUATOR. 2006 ANNUAL MEETING Arlington, Virginia. 10:00 - 11:15 am Friday, January 13, 2006. INDUSTRY / UNIVERSITY COOPERATIVE RESEARCH CENTERS PROGRAM. NATIONAL SCIENCE FOUNDATION. EVALUATING THE EVALUATOR.

bratcher
Download Presentation

2006 ANNUAL MEETING Arlington, Virginia

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluators' Breakout II: EVALUATING THE EVALUATOR 2006 ANNUAL MEETINGArlington, Virginia 10:00 - 11:15 am Friday, January 13, 2006 INDUSTRY / UNIVERSITY COOPERATIVE RESEARCH CENTERS PROGRAM NATIONALSCIENCE FOUNDATION

  2. EVALUATING THE EVALUATOR ERIC SUNDSTROMNSF IUCRC EvaluatorUniversity of Tennessee DENIS GRAYNSF IUCRC Evaluation ProgramNorth Carolina State University &

  3. Why This Session? • Questions from Center Directors at January, 2005, IUCRC meeting, and... • After all, we're Evaluators, it only figures that we'd eventually look at evaluating what we do...

  4. Goals for Today • Explore the problem of evaluating IUCRC Evaluators • Frame the problem, not attempt to propose a solution • Address 3 questions: 1) Who evaluates Evaluators? 2) Based on what? 3) Evaluate how? • Identify next steps

  5. 1. Who evaluates IUCRC Evaluators? • Evaluators ultimately report to NSF IUCRC Program Manager, Alex Schwarzkopf. • Evaluators also serve multiple customer-groups: - IUCRC Evaluation Program at NCSU (Denis) - Center Director & Site Co-Directors - Center Operations Managers / Administrators - Industry Member Representatives - Center Scientists & Graduate Students - Host University Administrators - Fellow IUCRC Evaluators

  6. Clarifying who evaluates IUCRC Evaluators: • What purposes does the IUCRC evaluation effort serve? • Who is the primary customer for the IUCRC evaluation?

  7. 2. Evaluate based on what? NSF and other counterparts can evaluate an IUCRC Evaluator's performance based on: • How the Evaluator performs the role: - Completeness of data-collection & reporting - Timeliness of data-collection, analysis, reporting - Quality of evaluation reports - Etc. • What the Evaluator role entails – evaluation Protocol: - Focus & methods of data-collection - Design & validity of questionnaires, other measures - Timing of data-collection, analysis, reporting - Structure & content of evaluation reports - Etc.

  8. Role of Evaluator - Main Elements • Program evaluation: Data-collection, analysis, reporting • Liaison & linkage: Communication with constituencies • Other duties, adopted or negotiated

  9. Role of NSF IUCRC Evaluator: Current Protocol • Program evaluation - Data-collection, analysis, reports: - Annual Evaluator's Report - Industry & FacultyQuestionnaires: Collect, analyze, report - LIFE Forms: Manage distributing, transcribing, IAB review - Exit interviews with Industry Member representatives • Liaison & linkage - Communication with constituencies: - Center Industrial Advisory Board meetings, Spring & Fall - Annual NSF IUCRC Program meeting, January - Annual NSF IUCRC Evaluators' meeting, June • Other duties, adopted or negotiated: - Consultant / Coach - Meeting Facilitator / Leader - Mediator, etc.

  10. How well does the evaluation protocol meet the needs of the “customer” (NSF)? • How important and useful for the customer is the information collected via the protocol? • Evaluator Report • Responses to Industry & Faculty Questionnaires • LIFE forms • Exit interviews • What factors affect the usefulness of the information? • Specific information collected • Burden of data collection • Sources of the information • Timeliness of feedback • How could the value of the information be enhanced?

  11. 3. Evaluate how? Evaluation of Evaluators incorporates content – basis or questions for evaluation – and a process or procedure. • Content: - Is the Evaluator doing the right job? (correct protocol) - How well is the Evaluator doing that job? (following protocol) • Process:Open question - Must support & align with the NSF IUCRC Program - Involves multiple counterparts - Other features?

  12. Next Steps • Review and evaluate the current Evaluator protocol • Consider re-designing the protocol • Explore options for revising the current role • Consider having Evaluators do self-evaluation via an Evaluator Practices Questionnaire • Others?

  13. Questions & Discussion?

  14. What could be done to improve the protocol? • Make evaluation tools shorter, more efficient and focused • Get higher response rates & make feedback to director quicker • Industry P/O  Industry feedback tool (one pager) • Provide LIFE form analysis: • Faculty P/O  Faculty feedback; switch to students after year 5 • Create alternative tools: focus group guides • Continue to encourage and support periodic research projects that would be implemented program-wide

  15. JUNE EVALUATOR MEETING June 14 - 16, 2006Holiday Inn Rosslyn / Key BridgeArlington, VA

  16. Goal for Today: Identify Preferences for... • Prospective speaker(s) • Session topics • Task Forces

  17. Prospective Speakers • MARIANN ("SAM") JELINEK - College of William & Mary, Williamsburg, VA. - Innovation, technology & organization; former Director of NSF's Innovation & Organizational Change Program. • DAVID ("DAVE") ROESSNER- SRI International, Arlington, VA. - Evaluation of Science & Technology; former NSF IUCRC Evaluator. • SCOTT SHANE - Case Western Reserve University - Innovation & Entrepreneurship; speaker at earlier IUCRC Evaluators' meeting. • BILL VALDEZ- Dept. of Energy, Washington DC, Director, Dept. of Planning & Analysis. - Founder, Washington Research Evaluation Network (WREN) consortium of R&D evaluators.

  18. Session Topics at June Evaluator Meeting? • Evaluation of R&D for Innovation, Advancing Technology, ... • Report of 2005 Process / Outcome Results & Trend Analysis • Evaluator Research: Results from Current Projects; New Projects • What Constitutes Success for a NSF IUCRC? (...for a consortium?) • New (Fine-tuned? Upgraded? Redesigned?) Evaluator Protocol • Evaluator's Report: Guidelines for Analysis (+ Recommendations?) • LIFE Form Procedure: Current Practice & Variations (& next steps?) • Evaluator Practices Questionnaire: Draft, process (& 1st Results?) • IAB Meeting Structure: Theme & Variations • Others?

  19. Proposed Evaluator Task Forces • Planning & Program for June Evaluator Meeting (Eric, Denis, ... & Kate Ryan) • Evaluator Protocol - Review & Update (Denis, David M., ... ) • Evaluator Research Projects & Agenda (Dave T., ...) • LIFE Form Practices - Current & Next (Otto, ... ) • Evaluator Practices Questionnaire & Process (Craig, Don, ... )

More Related