1 / 34

Choosing & Working With An Evaluator

Choosing & Working With An Evaluator. Welcome!!. We will begin approximately 2 minutes past the half-hour. Please MUTE your telephone . If you have no mute button, you may press *6 on your phone, or press the Mute Me icon on your screen.

guri
Download Presentation

Choosing & Working With An Evaluator

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Choosing & Working With An Evaluator Welcome!! We will begin approximately 2 minutes past the half-hour. Please MUTE your telephone. If you have no mute button,you may press *6 on your phone, or press the Mute Me icon on your screen.

  2. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Welcome Nan MaxwellSenior ResearcherMathematica Policy Research

  3. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Welcome Shanna JaggarsSenior Research AssociateCC Research CenterTeachers College, Columbia

  4. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Agenda Overview • Welcome • Nan Maxwell, MPR • Shanna Jaggars, CCRC • Writing an Effective RFP • Ann Person, MPR • Dawn Busick, Missouri CC Association • Negotiating with your Evaluator • Shanna Jaggars, CCRC • Chris McRoberts, NW Arkansas CC (Arkansas) • Creating an Effective Partnership • Debra Bragg, Office of CC Research & Leadership (U. Illinois) • DeRay Cole, Robeson CC (North Carolina)

  5. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Handling Q&A • Use the chat function to type in relevant questions at any point. • If you have answers / input on someone else’s question, feel free to type a response. • We will monitor the written questions and choose 1 or 2 to ask the presenters after each topic. • If time allows, we will also answer questions at the end more generally. • Key / common questions & answers will be summarized and sent out to all participants.

  6. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Guidelines for Preparing a Request for Proposals Ann PersonMathematica Policy Research TAACCCT Evaluation TA Webinar June 28, 2012

  7. The RFP: Answer basic questions COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • Who? Describe your consortium/program • Why? Describe what you want to get from the evaluation • What? Describe the scope of work for the evaluation • How? Describe terms of contract, proposal requirements, and vendor selection criteria • When? List proposal process dates and period of performance

  8. Who: Overview and Context COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • Describe TAACCCT grant program and its goals • Describe your consortium • Describe your program/theory of change • Inputs • Activities • Participation • Outputs • Outcomes

  9. Why: Statement of Purpose COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • What do you want to learn? • Implementation study • Outcomes study • Who are the primary audiences? • Institutions, systems • DOL • The broader field • How will results be used (by the different audiences)?

  10. What: Scope of Work COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • Design: • How will they define the comparison cohort? • How will they collect, aggregate, and analyze data for implementation/progress/outcome measures? • Will they need IRB clearance? How will they get it? • Activities • Key evaluation tasks • Key engagement tasks • Deliverables • If you’re unsure about specifics, describe what you want the end result to look like

  11. How: Terms of Contract COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • Roles and responsibilities • To whom will the evaluator answer? • How will they engage with different stakeholders? • Budget • Know what you can get for the funding you have • Don’t be coy about how much you have available • Your procurement procedures

  12. How: Proposal Requirements & Selection Criteria COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • Proposal Requirements • Statement of work • Vendor qualifications • Budget • Selection criteria • How much weight will you give to each section above? • Consider trade-offs (e.g., timing/cost/quality)

  13. When: Timeline COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • RFP process timeline • Period of performance • Deliverable dates (align with DOL?) • Activity/engagement dates • Gantt charts map activities and deliverables over time – you can develop one or include as a requirement of the proposal

  14. RFP Process: A few questions COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • How will you distribute your RFP and to whom? • How will you ensure a good range of applicants? • Salient procurement policies and legal requirements?

  15. Overarching Principles COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH • Be clear • Avoid jargon • Describe end result • Include supporting materials as appendices • Don’t make the vendor guess what you want – but rely on their expertise to figure out how to deliver on it

  16. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Writing an Effective RFP • Dawn Busick • Project DirectorMoHealthWINsMissouri CC Association

  17. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Writing an Effective RFP – Q & A Michelle HodaraPostdoctoral Research AssociateCC Research CenterTeachers College, Columbia

  18. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Negotiating with your Evaluator Shanna JaggarsSenior Research AssociateCC Research CenterTeachers College, Columbia

  19. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Clarify your learning goals • Least expensive, but also least informative

  20. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Clarify your learning goals • More expensive, but much more informative

  21. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Clarify your learning goals • Requires more staff time, but most useful in long run • Different evaluators will have different levels of comfort with / skill in handling these different types of goals

  22. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Explicit conversation about their needs/wants • Clarify differences between their needs and wants • Make clear which needs/wants you can meet • If you’re not able to meet needs, the evaluator cannot effectively do the job • If you’re not able to meet wants, evaluation may require more time, more money, or result in less useful evaluation. Adapted from Peter Block, “Flawless Consulting”

  23. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Example: Quantitative data needs vs. wants • Priority 1: Needed as soon as possible • Priority 2: Necessary, but can wait a while before receiving • Priority 3: Not absolutely necessary, but would be very helpful

  24. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Negotiating with your Evaluator Chris McRobertsExecutive Director,Path to Accelerated Completion & Employment (PACE)Northwest Arkansas CC

  25. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Negotiating with your Evaluator – Q & A Michelle HodaraPostdoctoral Research AssociateCC Research CenterTeachers College, Columbia

  26. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Creating an Effective Partnership:An Interview with Debra Bragg Debra Bragg Professor, Education Policy, Organization, and Leadership; Director, Office of CC Research & Leadership University of Illinois at Urbana-Champaign

  27. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH A good experience • Consider evaluation from the start • Openness • Curiosity • Communication • Trust

  28. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH A less-good experience • Mismatch of expectations • Not allowing process to evolve

  29. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Two key things • Allow evaluator to get to know leadership, colleges • Don’t panic – tap your evaluator’s knowledge

  30. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH TAA-specific advice • Identification / documentation / mapping of Program of Study • Understand data capacity of each college – don’t duplicate work

  31. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Creating an Effective Partnership DeRay ColeProject ManagerNorth Carolina Advanced Manufacturing AllianceRobeson Community College

  32. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Creating an Effective Partnership – Q & A Michelle HodaraPostdoctoral Research AssociateCC Research CenterTeachers College, Columbia

  33. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH General Q & A • “Raise your hand” if you would like to contribute to a question or answer • If Michelle calls on you, un-mute your phone to allow you to reply • When you are done speaking, re-mute your phone

  34. COMMUNITY COLLEGE RESEARCH CENTERMATHEMATICA POLICY RESEARCH Upcoming Events • Evaluator convening, August 7-8, probably in Chicago, Ill. • Follow-up webinar, probably September 12

More Related