Survey Research Basics

1. Survey Research Basics

2. Welcome from the FDI David J. Klocko, MPAS, PA-C Assistant Professor and Clinical Coordinator Department of Physician Assistant Studies UT Southwestern Medical Center at Dallas

3. How to get Audio • Turn on your computers speakers! • Use the Audio Setup in your control panel if you have any problems • If you don’t have speakers: • Dial: (312)-878-0211 • Audio PIN: Shown after joining the Webinar

4. Questions and Answers You can send us questions at any time Send your questions via the question section of the control panel If we do not get to your question, we will try to email a response

5. Introductions… Mei Liang, MS Assistant Director of Data and Research, PAEA mliang@PAEAonline.org Meredith Davison, PhD, MPH Chair, Research Institute Advisor Committee Associate Director, PA Program University of Oklahoma−Tulsa meredith-davison@ouhsc.edu

6. Let’s get started!

7. Survey Research Basics Meredith Davison, PhD, MPH Mei Liang, MS September 21, 2009

8. Overview of Surveys • Surveys are the most popular method of research • Definition: “a series of questions that are posed to a group of subjects with the intent of generalizing sample responses to a larger population.”

9. The Steps in a Survey • Establish the goals • Determine your sample • Choose the interview methodology • Create the questionnaire • Conduct the interviews • Analyze the data

10. Establishing the Goals • Important first step • Must limit the goals to be effective • What can you obtain: • Information • Opinions/Attitudes • Characteristics of populations • Evaluations

11. Selecting Your Sample • Target population • Sample size • Confidence interval • Confidence level – usually 95 or 99%

12. Confidence Measures • Confidence interval – the plus-or-minus figure usually reported in newspaper or television opinion poll results. • Confidence level – the level of certainty about the data; represents how often true percentage of population who would pick an answer lies within confidence interval. • With confidence levelandconfidence interval together, you can say that you are 95% sure that the true percentage of the population is between 43% and 51%. • The wider the confidence interval you are willing to accept, the more certain you can be that the whole population answers would be within that range.

13. Sample Size • Census vs. sample • Sampling • Simple Random Sampling • Clustering • Stratification • Sample calculation tools • Statistical packages • Online • The RI Website: http://www.paeaonline.org/index.php?ht=d/sp/i/80419/pid/80419

14. Sample Bias • Impossible to totally exclude bias • Types of bias: • Favorable – people extremely positive toward a certain view • Unfavorable – negative attitudes • Extreme views – outliers • Atypical opinions – variety of reasons • Non-working – daytime telephone

15. Interviewing Methods • Personal interviews • Individual • Focus groups • Telephone surveys • Mail surveys • Email surveys • Internet/intranet (web page) surveys

16. Personal Interviews Advantages: Personal interaction Know target audience Longer interviews may be tolerated Disadvantages: Expensive Time-consuming Potential for lack of honesty Non-representative sample

17. Telephone Surveys Advantages: Fast contact Dial random digits Software can be used Skilled interviewers can elicit answers, ask for clarification Disadvantages: Telemarketers have given legitimate research a bad name Can’t call during day, late at night, early morning, etc. Bias against cell-phone users and individuals without phones

18. Mail Surveys Advantages: Among least expensive Only type available if you have names and addresses Can include pictures Allow individuals to answer at their leisure Less intrusive Disadvantages: Time! Bias against populations with low literacy, immigrants, etc. Poor response rate (Incentives?)

19. Email Surveys Advantages: Very economical Very fast Pictures and sound files can be attached Novelty element may stimulate interest Disadvantages: Must have email addresses Some people may respond more than once People dislike unsolicited email survey Cannot generalize to general population Cannot automatically skip questions

20. Web-Page Surveys Advantages: Extremely fast Almost no cost Can show pictures Use complex question skipping logic, randomizations, etc. More honest answers? Longer responses Disadvantages: Internet surveys do not reflect general population People can quit in the middle of survey Anyone can reply Multiple responses

21. Questionnaire Design • Fit the questionnaire to the goals • KISS – short and simple • Do not add extra questions • Start with introduction, cover letter, etc. • Allow “Don’t Know” or “Not Applicable” (people do not like to feel coerced)

22. Question Types • Multiple choice • Single answer • Multiple answer • Numeric open end • Text open end • Rating scales • Agreement scales (Likert) • Branching questions

23. Multiple Choice Single answer Multiple answer Radio buttons Dropdown list (web) Check boxes Selection boxes (web)

24. Open Ended Numeric Text Be very specific of the format (%, $, or ,) Responses need a lot of editing Confirm with respondents when in doubt Use when you have to If necessary, specify the answer length Editing and grouping required for responses

25. Rating Scales Group similar items together Higher scale on the left (?) N/A option when necessary

26. Agreement Scale (Likert) “Neutral” option/midpoint is not a must have N/A option if needed

27. Branching Questions Q 2 Answer A Answers… Q 4 Q 1 Q 3 Answer B Answers… Pleases respondents Works especially well with web-based surveys Example

28. General Principles Early questions easy and pleasant Group questions on the same topic together Leave difficult or sensitive questions until near the end of survey Use logical or natural order to choices Always present “agree” “disagree”, not vice-versa. Be aware that question order can affect choices (Randomize answers if possible) Problem of habituation (change positive answer) Keep questionnaire as short as possible

29. General Principles (continued) Start with a title Use a logo/logos if possible Reassure about confidentiality Number pages of survey/Show survey progress (web) Include all relevant alternatives as answer choices Do not put two questions into one Avoid emotionally charged words Avoid technical terms and acronyms Leave demographic questions until the end?

30. Additional Tips for Web Surveys Make it attractive Use graphics sparingly Include an introduction or welcome Make page and question layout consistent Allow space for comment type questions Test it thoroughly Example 1; Example 2

31. Online Survey Software The tools are similar but different in nuances They offer different packages for different price levels Most services have a free version

32. Online Survey Software (continued) Survey tools easy to manipulate Manageable background Ability to upload graphics and videos Branching, piping, randomization, lookup tables, validation… Real-time data reporting ability Manageable email list Response tracking, automatic reminders

33. Pretest all Surveys Test on the same type of people you intend to sample Test run should reveal unanticipated problems with questions wording, instructions to skip questions, etc. If anything changes after pretest, you cannot combine the results

34. Common Survey Data Analysis Techniques Descriptive statistics – Mean, median, mode, percentiles, range Correlation’s – Spearman rank-order, Pearson product -moment Comparisons – Mann Whitney U, Chi-square, T-test, Analysis of variance Trends – Repeated measures of analysis, McNemar test

35. Selecting Appropriate Analysis Technique Appropriate analysis technique is dependent on the answer to the following questions: How many people are you surveying? Are you looking for relationships or associations? Will you be comparing groups? Will your survey be conducted once or several times? Are the recorded as numbers and percentages or scores and averages?

36. Motivating Respondents & Writing Cover Letters Necessary for all surveys, not just paper! Explain the purpose Describe the sponsor Offer information in exchange for responses Consider an advance letter Consider using other methods

37. Cover Letters Letterhead Date Name and phone number for further information Maximize attractiveness Enhance readability

38. Motivating Respondents Explain how to return the questionnaire Describe incentives Directly or indirectly provide a realistic estimate of the time required Explain confidentiality of data Specify survey length and expected time to finish survey Determine deadline date Have contact person Consider reminders

39. Final Words of Wisdom Expect a low response rate (5 – 40%) Remember that respondents have a strong tendency to exaggerate answers and give “correct” answers Discount “favorable” answers by a significant factor People give answers that they feel reflect well on them

40. Questions and Answers

41. This webinar is over! Thanks for attending! Please do notturn off your browser until you see the survey. We would appreciate it if you would complete the survey at this time. If you have any questions or suggestions please contact Toya Vigne at the PAEA National Office at tvigne@paeaonline.org If you have more questions about the webinar that has just ended, please email Mei Liang at mliang@paeaonline.org