Needs Assessment: Surveys

1. Needs Assessment:Surveys Dale A. Moore, DVM, PhD School of Vet Med, UC Davis

3. Objectives • Describe the needs for needs assessments • Describe one way to conduct a needs assessment • Provide some examples

4. Needs For What? • Program topics • Location • Delivery Method • Etc… A need is a discrepancy between what the desired goal or standard is with what actually exists.

5. Program Development • Identification of the Main Themes*** • Description of the Program Purpose • Conduct Needs Assessment*** • ID others interested in development & delivery of the program*** • Outline learning objectives*** • Select format and teaching methodologies*** • Select faculty / facilities*** • Evaluation tool*** • Marketing and promotion strategy*** / budget Nancy Bennett, Dev CPE Programs, 1990

6. Traditional CE • Lecture-dominated format • Little evidence of impact on behavior or outcomes • Minimal collaboration between learners and providers • Lack of timely response to learner needs • Emphasis on credit • Focus on course production driven by enrollment economy Don Moore, An approach to needs assessment in Physician practices. Alliance for CME, 1998

7. New Paradigm in CE • Increased emphasis on learning • Data-based educational activities • Collaborative learning systems • Blending of quality mgmt and CME • Focus on improving patient outcomes • CME as integral part of healthcare system Don Moore, An approach to needs assessment in Physician practices. Alliance for CME, 1998

8. The Need For Needs Assessment • ID learning objectives • Data-driven education • ID the standards of practice/knowledge • ID the goals for outcomes & stakeholder expectations • ID the role of the educational program in the practice of XXXXX

9. Needs Assessment for Better Evaluations! 1 1 1 1

10. Needs Assessment: Which Way to Go?

11. Different Levels of Needs • Societal needs / client / industry needs and expectations • Community needs • Profession’s needs / professional sector’s needs • Farm’s needs • Individual’s needs

12. Needs Assessment Methods • Environmental Scans • Review of literature • Discussion with experts • Survey • Mail survey • Telephone survey • Face-to-face interviews • Focus Groups • Clinical audits / Practice audits • Self-assessments

13. Surveys • Advantage of providing quantifiable responses from randomly-selected study participants • Usually require that the response choices are known by the investigator in advance • Issues of response-rates

14. List Frame for Surveys: WHO? • Your own/other database of previous participants • Company mailing lists • Others?

15. Selection From the Listframe • How to randomly select? • Random number generator in Excel • Tools/Data Analysis/Random Number Generator/OK/Variables…… • Sample size? EpiCalc 2000 • http://www.brixtonhealth.com/epicalc.html

16. Designing Your Survey and Survey Questions • Fowler, FJ. (1995). Improving Survey Questions: Design and Evaluation. Thousand Oaks: Sage Publications. • Dillman, DA. (2000). Mail and Internet Surveys: The Tailored Design Method. John Wiley and Sons: New York.

17. Online Surveys • ANR Site Builder • http://groups.ucanr.org/help/index.cfm • The mechanics of putting a survey together on-line • http://www.surveymonkey.com/

18. Improving Response Rates • Follow Dillman’s Method • Keep your promises to your constituents • Incentives – CASH • Multiple contacts • Respondent-friendly questionnaires

19. Improving Response Rates • The number of contacts Advance letters, postcards, follow-up, additional copies of questionnaires, telephone calls (5) • Salience of the topic Relevance. Common reason for non-response is that survey doesn't mean anything to the person.

20. Improving Response Rates • Government sponsorship: higher response rates than private organizations. • Special third contact Follow-up of advance letter / initial follow-up with special mailing procedures: certified mail or special delivery, personal or telephone contact. • Incentive on first contact: Incentives included with first mailing increase response rate. Incentives offered later not effective. ($5 cash)

21. What does good questionnairedesign involve? • • IDENTIFY MEASUREMENT OBJECTIVES! • • Draft questions based on objectives • • Test questions • – Expert review • – Cognitive interviewing • – Pilot run • • Format of questionnaire

22. Respondent-Friendly Q’aires • Not too long • Ask most important questions first; demographic information last • Unambiguous questions/responses • Easy to complete • Great directions -- subheadings

23. Critiquing/Drafting Questions • Consistently understood (avoid ambiguous words, define key terms) • Avoid “double-barreling” "(asking two questions at once) “Would you like to be rich and famous?” • Avoid hidden assumptions • Consistently administered and communicated – interviewers read all response options

24. Some Help With Questions • http://www.statpac.com/surveys/surveys.doc • http://www.leeds.ac.uk/iss/documentation/top/top2.pdf • http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm#rsi • Many others and books, like Dillman

25. 9 rules for writing good questions • 1. Remember your survey's purpose • 2. If in doubt, throw it out3. Keep your questions simple

26. Example: • "Imagine a situation where the production supervisor is away from the line, a series of defective parts is being manufactured, and you just heard that a new client requires ten thousand of these parts in order to make their production schedule. How empowered do you feel by your organization to stop the line and make the repairs to the manufacturing equipment?"

27. 4. Stay focused - avoid vague issues"When did you last see a movie?" • 5. If a question can be misinterpreted, it will be"What time do you normally eat dinner?" • 6. Include only one topic per question

28. 7. Avoid leading questions "Most doctors believe that exercise is good for you. Do you agree?" • 8. Consider alternate ways to ask sensitive questions Income, drug or alcohol consumption and sexual habits. • 9. Make sure the respondent has enough information

29. 5 rules for obtaining usable answers • 1. Response options mutually exclusive and exhaustive. • 2. Keep open-ended questions to a minimum they pose problems in terms of coding and analysis. • 3. People interpret things differently, particularly when it comes to time Trouble-spots include responses such as "Always," "Sometimes" and "Never." Build in a temporal frame of reference. "I am going to read a list of publications. For each one, please tell me whether you read it regularly. By regularly I mean, at least three out of every four issues."

30. 4. Consider a "Don't Know" response Allow people to say they do not have an opinion. • 5. Provide a meaningful scale End points must be anchored with labels. “Rate your satisfaction where 1 means 'Very Satisfied' and 5 means 'Very Dissatisfied.‘ Odd number of points provides middle alternative. Gain nothing by having scale with > 7 points.

31. Survey Length • 30-40 questions • Fewer the better • Booklet form?

32. Survey Flow

33. Pilot-Testing • With peers • With a small group of your audience • Helps refine the tool

34. Surveys for Needs Assessment • What are your outcomes? • What is your audience? • Sampling frame, sample size, how to select randomly • Survey distribution • Questionnaire design • Data analysis