450 likes | 461 Views
Learn essential tips for conducting evaluation research effectively, from setting objectives to choosing the right survey methods. Helpful insights for DIY researchers and clients alike.
E N D
David Geddes, Ph.D. Senior Vice President & Partner Fleishman-Hillard Institute for Public Relations Measurement Commission 21 tips for conducting evaluation research on a boot lace
Today’s objectives • Advice on doing it yourself • Advice on being a good client
Where to get help • University departments • Social work, Sociology, Psychology, Political Science, Business, and others • Contacts via board of directors • American Marketing Association • Market Research Association • Market research firms • LinkedIn (www.linkedin.com) • National Opinion Research Center (University of Chicago) http://www.whatisasurvey.info/
http://www.italladdsup.gov/community_partners/dc_teachmehow_02.asphttp://www.italladdsup.gov/community_partners/dc_teachmehow_02.asp • Introduction to evaluation research • Process evaluation examples • Telephone survey with annotations • Intercept and self-administered survey
Build a community • Vision • Lutheran Social Services Research Community • Share best practices • Methods • Facebook or other collaboration tool
1 Set your objectives • Program objectives • Specific information needs
2 Do your background research
3 Think about different types of evaluation and measurement • Formative evaluation • Helps planning • Process evaluation • How well is the program operating? • Impact evaluation • Building awareness and understanding? • Outcome evaluation • Concrete program or “business” metrics
4 Don’t assume you know audience attitudes and opinions • You need formative research
5 Set up your process metrics
6 Measure impacts • Awareness • Knowledge • Attitudes and perceptions • Intended actions
Known respondents Population =Sample frame Members Donors Clients Employees Civic organization leaders Defined population Sample frame derived from population General public Congregation members Potential donors Potential adoptive parents 7 Identify survey targets
8 Pick the right survey method to reach targets • Telephone • Mail • Online • Mall or store intercept
Telephone surveys • Advantages • Speed, in some cases • Control • Probe and clarify • Cost if you do it yourself • Disadvantages • Cost if you go outside • Hard to reach some socio-economic groups
Telephone surveys • Case: General public, 10 minute survey of 500 adults in Saint Louis metro area. • $8,000 = $16 cost per interview • Review cost parameters • Case: Survey of 200 adult children or guardians of seniors who receive services from LSS in St. Louis metro area. LSS provides names and phone list. • $4,000= $20 cost per interview • Review cost parameters
Telephone surveys • Case: Survey of 200 married adults in the St. Louis metro area aged 18 to 45 who are at least somewhat open to the idea of adoption. 10 minute survey. 30% incidence based on other surveys. • $7,000 = $35 cost per interview
Doing your own telephone surveys • Sample list • Staff • Interns, volunteers, students, etc. • Call management system • Data recording system • Discipline • Persistence
Going outside for telephone surveys • Smaller local firms • Be flexible on timing
Mail surveys • Advantages • Lower cost • Easy if you have a mailing list • Easier to do yourself • Disadvantages • Slow • Lower control • Lower response rates • Need a mailing list
Doing your own mail surveys • Clear questionnaire layout • Sample list • Staff • Interns, volunteers, students, etc. • Lots of manual paper handling • Discipline • Patience
Online surveys • Advantages • Speed, in some cases • Control • Cost if you do it yourself (SurveyMonkey) • Disadvantages • Purchase a sample list • Unrepresentative of some socioeconomic groups
Online surveys • Case: General public, 10 minute survey of 500 adults in Saint Louis metro area using an online survey tool. • $3,300 = $6.60 cost per interview • Review cost parameters
Doing your own Online surveys • SurveyMonkey programmer • Sample list • Staff • Interns, volunteers, students, etc.
Mall intercept surveys • Advantages • Control • Probe and clarify • Cost if you do it yourself • Disadvantages • Slow • Need mall or store cooperation • Unrepresentative of some socioeconomic groups
Doing your own mall intercept surveys • Cooperation of mall or Wal-Mart • Staff • Interns, volunteers, students, etc. • Discipline • Patience .. Figure on one interview/hour
8a Select an appropriate sample size • What are possible sources of error? • How far off could we be? • What sample size do we need?
9 Give advance notice • If you know the target respondents • Newsletter • E-mail • Postcard • Benefit: improves response rate
10 Remember you are asking for their time • This applies to… • Pre-survey letters or cards • Introduction to the survey • And the questions themselves • Don’t ask meaningless questions
11 Write a good introduction to the questionnaire • Introduce yourself • “Research” not “survey” • Survey topic: social services • Not a sales call … not a fundraising call • You want their opinion / input
12 Move from general to specific
13 Lead with an interesting question • Engage the respondent • Ask interesting questions and people will talk
14 Make judicious use of open-ended questions • Example: • “Based on what you have read or heard, what are your personal perceptions of Lutheran Social Services? [PROBE] Anything else?” • 2 to 3 per questionnaire maximum • Finish with a “final thoughts” question
15 Use good response scales • Building a scale • Clearly distinguish good and bad • 1 Agree • 2 Disagree
Use good response scales • Building a scale • Add a polite positive and negative • 1 Strongly Agree • 2 Agree • 3 Disagree • 4 Strongly disagree
Use good response scales • Building a scale • Add a top positive response • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Disagree • 5 Strongly disagree
Use good response scales • Building a scale • Add a don’t know/no opinion • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Disagree • 5 Strongly disagree • 6 Don’t know / no opinion
Use good response scales • Building a scale • Add a mid-point if you really need it ... and balance the scale • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Neither agree nor disagree • 5 Disagree • 6 Strongly disagree • 7 Very strongly disagree • 8 Don’t know / no opinion
16 Make sure you cover all possible responses • Multiple choice questions
17 “No opinion” does not equal “Don’t know”
18 Always include a “refused” option
19 Be careful with numeric responses • “About how much do you spend on gas each month” $__________ • Which of the following best describes the amount you spend on gas each month: • Under $25 • $26 to $50 • $51 to $100 • Over $100
20 Be careful about “do you floss your teeth” items • Socially desirable responses • Responses that may imply guilt
21 Test your questionnaire
Wrap-up • Begin with your objectives and information needs • Define the people you want to survey • Select an appropriate survey method • Write good questions • Test • Launch the survey • Analyze the results