PPA 502 – Program Evaluation

1. PPA 502 – Program Evaluation Lecture 5a – Survey research

2. Introduction • Government administrators and elected officials love to claim that they possess a profound understanding of their public’s needs, desires, and disaffection.

3. Introduction • Unfortunately, the administrators and officials are learning that storms of controversy provide meager evidence of the workaday values of the everyday people they govern. • Surveys of the public, conducted following the basic precepts of survey design and analysis, are fast becoming the vehicle for genuine connection to the public will.

4. Introduction • Uses. • Evaluations of government services. • Changing demographics that may signal shifts in service demand. • Patterns of service utilization. • Problem identification. • Customer service.

5. Introduction • Surveys have several important qualities. • Anonymity to respondents. • Point of view, characteristics, or use patterns can be characterized with little confusion. • Good surveys provide input from a representative cross-section.

6. Begin Before the Beginning • The best surveys grow from well-conceived and well-articulated reasons for conducting them. • Resist the temptation to hit the ground running. • Be certain of the purposes of the survey.

7. Begin Before the Beginning • Identify the appropriate audiences. • Identify the political and personal will for doing the survey. • Determine whether the questionnaire to be developed is better as a one-time or periodic survey. • Think about the usefulness of comparative data.

8. Getting Started • Convene a steering committee with key stakeholders. • Enlist the help of top government officials or administrators.

9. Designing the Survey • Sampling. • Choose the appropriate sampling frame: about what population do you wish to generalize? • A sampling plan must give every respondent in the sampling universe an equal chance of ending up in the sample. • Simple random sample. • Stratified sampling. • Stratified random cluster sampling.

10. Designing the Survey • Targeting the individual in the household. • If no list exists, you may only have addresses or phone numbers. If so, use household member with most recent birthday.

11. Mail, Phone, or In-person Interviews • The best ways to conduct surveys vary by accuracy, speed, and cost. • Most common are mail and phone surveys.

12. Mail, Phone, or In-person Interviews

13. Mail, Phone, or In-person Interviews • Increasing response rates. • Multiple mailings (up to three) with stamped, return address envelope. • Press coverage. • Combination of methods often best: Mail survey with telephone and in-person followup.

14. Mail, Phone, or In-person Interviews • Selecting sample size. • The size of sample depends on desired precision of estimates. • Generally speaking, if opinions are split as much as possible, than 100 residents will have a margin of error of +/- 10% with 95 percent confidence. Four hundred residents the margin is +/- 5%. • In general, 100 is a good minimum number, especially for subgroups.

15. Questionnaire Construction • Each question should be judged against the purposes of the survey and the uses to which it will be put. • Steal widely. • National Citizen Survey from International City Management Association and National Research Center.

16. Questionnaire Construction • Major principles. • Consistency. • Clarity. • Vague wording. • Double-barreled questions. • Assumed knowledge. • Overlapping response categories. • Simplicity. • Specificity. • Brevity (30 min. Phone, 60 min. In-person, 10 page mail). • Context sensitivity.

17. Questionnaire Construction • Major principles. • Security. • Demographic at end. • General to specific. • Fairness. • Option symmetry (balanced responses). • Option wording and order. • Background info, pros and cons, opinion. • Randomize pros and cons in a complicated survey.

18. Conducting the Survey • The survey steering committee. • Double check questionnaire with steering committee. • Frequency of surveys. • For most multipurpose surveys, no more than once per year. • Pretest. • Test on twenty people at random. Ask questions about format and clarity.

19. Conducting the Survey • Training. • Survey assistants must be trained. All must operate uniformly, asking the questions in the same way, coding in the same way. • Consistent open-ended coding. • 10% recontact of survey respondents. • Trying hard and keeping track. • Three contacts by telephone for each number. • Warning and at least two mailings for mail surveys.

20. Reporting Results • Data analysis. • For most government surveys, percentages, average responses, simple cross-classifications. • The most complicated analysis will be to get accurate population estimates – weighting. • Report writing and presentation. • Executive summary. • Bulleted lists. • Document survey methods in appendix. • Augment tables with bar and pie charts. • Powerpoint for in person presentation.

21. Hiring a Consultant • Previous experience. • Ability to communicate findings. • Share work with in-house staff. • Intuition.