I ntroduction to S urvey Q uality

1. Introduction to Survey Quality Paul P. Biemer RTI International Lars E. Lyberg Statistics Sweden

2. Course Content • Concepts of data quality • Survey measurement process • Coverage error and nonresponse • Survey instrument • Interviewers and interviewing

3. Mode and Setting • Data processing • Evaluation methods • Practical implications

4. The Evolution of Survey Process Quality Chapter 1

5. Concepts • Survey • Survey methodology • Quality • Survey quality dimensions • Survey process quality

6. Quality assurance • Quality control • Error sources • Mean squared error

7. The Concept of a Survey • concerns a set of objects comprising a population • population under study has one or more measurable properties • goal is to describe the population by one or more parameters defined in terms of the measurable properties

8. The Concept of a Survey (con’d) • access to the population requires a frame is needed • sample is selected in accordance with a sampling design specifying a probability mechanism and a sample size

9. The Concept of a Survey (con’d) • observations are made in accordance with a measurement process • based on the measurements an estimation process is applied to compute estimates • purpose is to infer to the population

10. Typical Shortcomings • target population is changed during the study • selection probabilities are not known for all selected units • correct estimation formulas are not used

11. Types of Surveys • One-time • Repeated or continuing • The survey environment • The survey infrastructure

12. A Brief History • Biblical censuses • Political arithmetic 1650-1800 • The 1895 ISI proposal regarding representative investigations • Bowley argues for random sampling 1913

13. The 1934 Neyman paper on the representative method • Neyman develops theories for sampling and confidence intervals • Nonsampling error theory in the 1940s

14. Interpenetration • The US Census Bureau survey model • Developments in other disciplines • Questions and interviewers • The response process

15. The Quality Revolution • Deming’s 14 points • Juran’s spiral of progress • Ishikawa’s 7 quality control tools • Joiner’s triangle (quality, scientific approach, teamwork)

16. Shewhart’s control chart for process control • Dodge and Romig’s acceptance sampling • A theory for statistical process control

17. Definitions of Quality • Fitness for use • Quality of design • Quality of conformance

18. Quality dimensions in official statistics (one of them is accuracy) • Quality according to some business excellence model • Performance indicators

19. Eurostat’s Quality Dimensions • Relevance of statistical concepts • Accuracy of estimates • Timeliness and punctuality in disseminating results • Accessibility and clarity of the information

20. Comparability • Coherence • Completeness

21. The Process View • Product characteristics are established together with the user • The quality of the product is decided by the processes generating the product • The processes are controlled via key process variables

22. Measuring and Documenting Quality • Accuracy can be measured • Other quality dimensions are qualitative and can be seen as constraints • Quality profiles • Quality reports • Performance measures

23. Examples of Tools • Self-assessment via excellence model • Checklists • Quality management • External auditing • Customer satisfaction surveys

24. Improving Quality • Changing processes • Project teams • Standardization via current best methods documents • Development of quality guidelines

25. We Concentrate on Accuracy • Data must be of sufficient quality for decision-making • Other dimensions are constraints • Accuracy is much more difficult to understand • It is important to convey information on error sources and their contributions to total survey error