1 / 21

Frequently Asked Evaluation Questions

Frequently Asked Evaluation Questions . Tobacco Control Evaluation Center Jeanette Treiber, Robin Kipke and Travis Satterlund April 28, 2011. Where Do We Start?. Each evaluation activity is derived from the plan: Objective plan type  outcome/process data

rhoswen
Download Presentation

Frequently Asked Evaluation Questions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Frequently Asked Evaluation Questions Tobacco Control Evaluation Center Jeanette Treiber, Robin Kipke and Travis Satterlund April 28, 2011

  2. Where Do We Start? Each evaluation activity is derived from the plan: • Objective plan type  outcome/process data • How will data be used to move the objective forward? • Who should the data inform or convince? • Map out what pieces of information are needed, and how best to collect it (method, from whom, where, when)

  3. What Is End-use Strategizing? • Identify your evaluation goals • Know the kind of information needed and specific purpose it should serve • Understand the best source for your data • Figure out when the data should be collected • Recognize what type of method is most useful • Decide an appropriate sample size

  4. How Can We Use Our Data? Your data can inform project staff by: • Measuring how your message is being perceived by various stakeholders (subgroups) • Indicating whether further community education efforts are needed • Assessing which policy options to pursue • Identifying where your campaign might encounter opposition • Signaling that perhaps it is not yet the right time to pursue policy

  5. Are There Other Uses For Our Data? Through a variety of reporting strategies, programs can use data to: • Indicate public/stakeholder readiness for a policy to decisionmakers • Generate “buzz” by sharing results in meetings, presentations, fact sheets, press releases • Build a groundswell of support and motivate more people to join the effort • Counteract critics

  6. Should We Create New Instruments? Adapt existing instruments (based on end use strategizing): • Not always necessary to reinvent the wheel • Adapt other data collection instruments for your project’s purposes • Start with high quality instrument • Add questions you need data on • Delete unnecessary questions • Make sure it will collect what you need

  7. Where Can We Find Instruments? • TCEC website has model instruments on limited number of objectives • Contact TCEC to access repository of instruments created/used by other projects • Ask other projects working on similar objectives to share their instruments

  8. What Other Sites Have Instruments? Download instruments from statewide sites • CA Health Interview Survey – http://www.chis.ucla.edu/questionnaires.html • CA Healthy Kids Survey – http://www.wested.org/cs/chks/view/chks_s/17?x-layout=surveys • CDC – http://cdc.gov/tobacco/data_statistics/surveys/ • YTPS on STORE Website – http://www.tcsstore.org/stages/1_document/t_yps.pdf

  9. How Can We Reach a Large Sample? • Utilize high traffic venues/locations • Back-to-school packets • Church service programs • Grocery store flyers • Classroom teachers • Entertainment tickets (e.g. movie theaters) • Organizational newsletters • Fairs and local events

  10. What Are Other Distribution Avenues? • Online surveys (SurveyMonkey) • Invite participation via email, social networking media (Facebook, Twitter) • Your organization’s website (and those of coalition members, agencies that serve target population)

  11. What Are the Pros of Online Surveys? • Cost effective • Faster delivery • Quick response time • Better at addressing sensitive issues • Ease of layout design and analysis

  12. How Can We Increase Response Rates for Online Surveys? Some tips for maximizing the response rates: • Request participation in advance • Include a cover letter • Allow sufficient time frame to complete the survey • Provide survey instructions • Design a clear survey • Send reminders to those who still have not completed the survey • Offer incentives for completion

  13. Do We Need to Pilot Test the Instrument? • Evaluate instrument’s effectiveness • Get a sense of how long it takes to complete • Create checklist of concerns to address with the pre-test • Do respondents/data collectors understand the instrument? • Is the wording clear to all segments of sample population? • Do any items produce irritation or embarrassment? • Have important issues been overlooked?

  14. What Should Our Key Informant Interview Protocol Look Like? • Identify your goals beforehand • Know the kind of information needed and specific purpose it should serve • Create a structure to your interview • Probe, probe, probe (write probes into your interview protocol)

  15. What Kind Of Outcome Data Do We Need For A MUH Objective? Depending on the objective: • Observed presence and placement of signage • Observation data of smoking and tobacco litter • Confirmed smoke-free language in leases • Tabulated count of signed leases with smoke-free language

  16. How Can We Tap Into Volunteer Labor? • Develop relationships with local universities, high schools, programs or clubs • Recruit students looking for research experience to do data collection • Offer training, internships, course credit, other incentives • Entice coalition members to join in the activities

  17. How Do We Make Use Of Our Volunteers? • To field test instruments/materials for appropriateness to data sources • To translate instruments/materials into other languages • To collect data • To provide access to other distribution points

  18. How Do We Write Up the Results From Our Data Collection? • Summarize as you go • Write “mini-analyses” • See if the data can help you meet your objective • Incorporate your summaries and analyses into your reports

  19. How Can We Utilize TCEC’s Resources? • Online coalition satisfaction survey administration and analysis service • Additional model data collection instruments on a variety of topics will be developed in coming months • Model instruments translated into other languages (Spanish, Vietnamese, Hmong and Russian) • We’re working with other statewide projects to develop tools that will help you use and share your data more effectively for your policy work

  20. When Should We Contact TCEC? For individualized assistance with: • evaluation planning • development of data collection instruments • advice on analyzing & interpreting data • guidelines for reporting & sharing info

  21. Resources You Can Tap • TCEC archive of data collection instruments • Evaluation associates at TCEC who can advise you on developing instruments • Recorded webinars on survey development, sampling decisions, conducting POS • Tips & Toolson creating interview questions, sampling and end-use strategizing • Electronic quarterly newsletter • Your colleagues working on similar objectives

More Related