Partnering with an Evaluator Presented by Joanne Kahn

Partnering with an Evaluator Presented by Joanne Kahn

Session goals • Understand how to hire and partner with a consultant to assist you with evaluation. • Set the stage for an effective evaluation.

Steps in the partnership 1 - Project definition 2- Vendor selection 3- Contract negotiation 4- Data collection 5- Reporting

Project Definition: what to include? Things you could do in-house: • Library use statistics • Point-of-use inquiry by staff • Skills tests • Feedback forms/surveys (but could get design help) Outside firm helpful for: • Survey design • Focus groups • Interviews • Observation • Instructor assessments

Steps in the partnership Project definition Vendor selection Contract negotiation Data collection Reporting

Vendor selection Does your library have an existing relationship with a particular research firm? To find an evaluator, ask peers or check the American Evaluators Association website www.eval.org. Members commit to: • Systematic inquiry • Competence • Integrity/honesty • Respect for people • Responsibilities for general and public welfare

Vendor selection Interview for: • Compatibility with your goals • Flexibility • Experience with methods you want to use • Staff who speak/write in needed languages • Schedule availability • Willing to work within your budget constraints • A portfolio of similar work • References — are they easy to work with? See handout: article from “Usable Knowledge.” www.usablellc.net/White_Papers/Hiring%20an%20Evaluation%20Consultant.pdf

Points to discuss with the evaluator • Timing and frequency • Who will do each task? • Which participants? How many? • Protecting the confidentiality of participants’ personal information • How will participants be recruited? • Incentives and perhaps transportation for participants

Evaluator’s perspective* • Focus on the story you want to tell stakeholders, and collect the data you need to tell it. • If you change staff (or delegate), make sure everyone knows the goals. • Plan how you will handle unexpected or negative feedback. The evaluator will take the approach that all information is useful. *Special thanks to Juliette Mackin of NPC Research.

Contract negotiation • Ask for a written proposal and fixed-price bid, and understand how the total was reached (hourly rates and line-item costs). • If bid exceeds budget, look for ways to reduce costs: • Junior staff members may be less expensive than partners • You may be able to reduce their time by: • Arranging the venue, incentives, and refreshments • Providing pre-screened participants with phone/email contact info • Bundling this project with another

Contract negotiation • Set a schedule that includes time for review and revisions. • Agree up front: • What intermediate deliverables will be provided? • Who needs to sign-off on the report?

Data collection — surveys • You may draft questions for the evaluator to review and include. • Have you made the process as participant-friendly as possible? • Measure before/after results. • Are the questions clear? For multiple choice questions, are the choices adequate? • Are there open-ended questions?

How will you recruit/screen participants? Have you made the logistics as participant-friendly as possible? The evaluator will probably want you out of the room. Review script for completeness, especially follow-on questions. Will there be a transcript or recording? Data collection — focus groups

Reporting Respect the firm’s objectivity! But it’s fair to ask them to have a positive outlook.

Reporting: make it readable! Executive summary conveys the main points Outline is easy to follow Visually attractive – photos!! Do tables/graphs improve the report? Are patron comments or anecdotes included? Patterns — do data sources corroborate each other? Conclusions should follow from the data

Partnering with an Evaluator Presented by Joanne Kahn