Data Burger: Response Rate Improvement and Basic Quantitative Data Analysis

1. Data Burger:Response Rate ImprovementandBasic Quantitative Data Analysis Nikki Dettmar National Network of Libraries of Medicine Outreach Evaluation Resource Center July 22, 2014 This webinar is adapted from classes created by Cynthia Olney, PhD, NN/LM Outreach Evaluation Resource Center

2. Today’s Topics • Response rate overview: • What is a ‘good’ response rate? • Maximizing response rate • Assessing response rate • Basic quantitative data analysis • Descriptive statistics • Levels of data • Graphs and charts

3. (Adapted from a class created by Cynthia Olney, PhD, Outreach Evaluation Resource Center) Part One: Response Rate Overview

4. Why use a questionnaire? • Learn about characteristics of a group • Evaluate quality of programs and services • Document results of programs and services

5. OERC Resources Questionnaire Design • Details about questionnaire design: http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st1 • Webinar recording: https://webmeeting.nih.gov/p90742547/ • “Writing Good Questions” handout: http://nnlm.gov/pnr/training/Writing_good_questions.pdf

6. Questionnaires primarily gather quantitative data • Exception: • Open-ended questions gather qualitative data • Information about basic qualitative data analysis is here: http://nnlm.gov/evaluation/booklets508/bookletThree508.html#qual

7. Three goals when using questionnaires

8. Why? “A low cooperation or response rate does more damage in rendering a survey's results questionable than a small sample, because there may be no valid way scientifically of inferring the characteristics of the population represented by non-respondents.” Goal 3: Get a high rate of participation from respondents (response rate) American Association of Public Opinion Research, 2002, Standards and Best Practices

9. Response rate defined Equation # of completed and partially completed questionnaires # of eligible participants in your sample

10. What is a “good” response rate? A rule of thumb from a standard textbook: • 50% is adequate • 60% is good • 75% is very good The Practice of Social Research. Earl R. Babbie. Belmont, Calif : Wadsworth Cengage, 2007.

11. Standard 5-day 1997: 36% 2003: 25% Rigorous 21 weeks 1997: 61% 2003: 50% Response rates have been declining Pew Research Center Experiment* http://poq.oxfordjournals.org/cgi/content/full/70/5/759 *Telephone Survey

12. Dillman has found the following principles increase response rates Dillman, et al., Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, Third Edition. Hoboken: Wiley, 2009.

13. Here’s how to… Decrease cost

14. Here’s how to… Increase trust

15. Improving response rates Improving response rates • Short surveys (1-2 pages) • Special third contact (e.g., certified mail, telephone call) • Content meaningful to respondents • Government sponsorship (as opposed to corporate or marketing firm) • Survey population (employee, school, military) • Pre-paid incentives (cash works best) http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2

16. Surface mail more effective than email for healthcare providers Effective Mail Procedure http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F5 Klabunde, CN, et al. “Facilitators and Barriers to Survey Participation by Physicians: A Call to Action for Researchers.” Evaluation & the Health Professions 2013 36(3):279-295. Adapted from: Don A. Dillman, Washington State University January, 2010

17. Example of email/online administration http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2 Adapted from: Don A. Dillman, Washington State University January, 2010

18. Tips for pre-notification letters Improving response rates Pre-notification letter/email: • Signature that respondents know and trust. Also, for electronic questionnaires, use that person’s name and email address in the “FROM” field if possible. • Briefly describe the project and why it is important that they respond. (Emphasize how it is important to them as well as how their responses are important to you and your organization.) • Explain when and how they can expect the questionnaire and who will send it to them. • Thank them in advance for their help. http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6

19. Tips for cover letters Cover letter/email: • Briefly describe your project and why you want respondents to reply to your questionnaire. • Include a motivational appeal and explain how to complete the questionnaire. • Describe who will see their individual responses and how you will maintain confidentiality and any risks to the respondent if he or she chooses to respond. • Describe incentives. http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6

20. Tips for reminder letters Checklist for reminder letters/emails: • State that you are sending a reminder for them to complete the questionnaire. • Thank those who have already responded. • Request those who have not responded to do so by a given date. • For electronic questionnaires, include a link to the questionnaire in every follow-up. http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6

21. Incentives work better when sent before or with the questionnaire Cost of Obligation Cost of Motivation Obligation is cheaper than motivation

22. Incentives: cash is best • Prepaid cash had strongest effect • Contributions to charity had little effect • Lotteries had no effect Warriner K, Goyder J, Gjertsen PH, McSpurren, K. Charities, no; lotteries, no; cash, yes: Main effects and interactions in a Canadian incentives survey. Public Opinion Quarterly, 1996, 60(4), 452-563 3

23. Bigger incentives are not necessarily better Adapted from: Don A. Dillman, Washington State University, January, 2010

24. Summary: Maximizing response rates • Use time-tested methods to boost response rates • Check for bias regardless of response rates • Use more than one source of information to assess the accuracy of your findings http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2

25. Use some key tools to assess response rates

26. More tools to assess response rates • Purposeful sampling: quota sampling • Triangulate (corroborate) - collect information from multiple sources and look for consistency

27. (Adapted from a class created by Cynthia Olney, PhD, Outreach Evaluation Resource Center) Part Two: Basic Quantitative Data Analysis

28. You probably have been – or will be – called to present a "case" for your programs

29. It helps to approach data analysis in an organized way. • Gather data (use credible procedures) • Summarize (reduce and display data) • Apply findings (pass judgment & make decisions) http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st3

30. There is a process that will help you analyze and use data • Learn the logic of evaluation • Use data to describe what happened • Judge value • Make decisions

31. Step One: Learn the logic of evaluation

32. Evaluation is a logical process • 1. Establish criteria • 2. Construct standards • 3. Describe what happened • 4. Compare with standards • 5. Make judgment • 6. Make recommendations

33. Step Two: Describe what happened

34. Data analysis starts with gathering your data Trees Forest

35. Evaluations usually require analysis of quantitative and qualitative data Today’s focus Qualitative methods for text Quantitative methods for numbers http://nnlm.gov/evaluation/booklets508/bookletThree508.html#ste3

36. Excel is a good program to use for gathering your data.

37. Handy Excel tip: "Format Cells" feature helps you format data Use Excel Functions to perform calculations for you. For this, I used the COUNT function: =COUNTIF(A2:A1675,“x")

38. Handy Excel tip 2: "Format Cells" feature helps you format the data Using your mouse, “right-click” to display this menu to format a number as a percentage.

39. Descriptive statistics summarize distributions of numbers • Frequencies • Percentages • Charts • Measures of Central Tendency (mean, median, mode)

40. Frequency distributions show patterns of response Would you attend an instructional session on Survey Monkey? (N=100)

41. Frequencies help to "clean" data and check for vulnerabilities Outlier responses Socially desirable responses Low or high response rate to response options Low response rate for certain questions

42. Charts show pictures of frequency distributions

43. Bar charts display comparisons across categories Would you attend an instructional session on Survey Monkey?

44. Bar charts can compare subgroups Do you need easy-to-read patient information In the following languages?

45. Line charts are particularly useful for showing change over time

46. Line charts also can show changes over time for multiple factors

47. Pie charts show sub-group breakdown of a larger group (demographics) In which type of library do you work? (check one)

48. Visual comparisons in pie charts are less obviousthan in bar charts In which type of library do you work? (check one)

49. Measures of central tendency are the "most representative" number in a data set • Mean (average) • Median (middle) • Mode (most frequent)

50. The mean (average) is the sum of all scores divided by total number of responses