330 likes | 487 Views
Designing for Reality. QMSS2 Seminar 3 on Survey Quality Leinsweiler , Dec. 2010 Peter Ph. Mohler. Doing surveys is a privilege and great fun!. Overview. Prolegomena Designing for Reality Conclusion . Prolegomena. Social Surveys are very, very robust measurement devices
E N D
Designing for Reality QMSS2 Seminar 3 on Survey Quality Leinsweiler, Dec. 2010 Peter Ph. Mohler
Doing surveys is a privilege and great fun! QMSS2 Peter Mohler 2010
Overview • Prolegomena • Designing for Reality • Conclusion QMSS2 Peter Mohler 2010
Prolegomena • Social Surveys are very, very robust measurement devices • Screening of ALLBUS showed over a period of 20 years only one ‘deviation’ (due to a mis-design in item sequence • Exception: web-panels of sorts used in marketing due to an extreme fluctuation of the panel participants QMSS2 Peter Mohler 2010
Prolegomena • This seminar discussed “problems” on a very high quality and precision level reflecting the needs of government agencies QMSS2 Peter Mohler 2010
Prolegomena • Accuracy/Precision/Micro-census Benchmarking • The quality of surveys is often defined by checking core demographics using a Micro-Census as benchmark • For instance Age, Gender, Education, or Regional Distribution • Three surveys • A) Personal address, age, gender, citizenship from residents lists • B) Like 1, but more calls per respondent • C) Address of household with substitution on the primary sampling unit (PSU) level by interviewer QMSS2 Peter Mohler 2010
Prolegomena - Benchmarking • Survey C is best • Note Micro-Census is also a survey with MSE! QMSS2 Peter Mohler 2010
Reality Check – Why to Break Your Neck? • Survey A and B cost much, much more than Survey C (including exhausting interviewers and staff) • Survey A and B need more time for data collection • So, why should you break your neck, when you can get better benchmarking by less effort and costs? (Attention: Herecy!) • However, the real question is, which effort and costs for what survey? (MSE) QMSS2 Peter Mohler 2010
Analysis in a Real World • Marketing • 20% vs- 80% market share • Election polls • +/- 2% precision for predicting election outcomes • Health prevalence • 1% point estimate error may result in millions of dollars mis-allocated • Comparative research • 1 minute lost due to mis-localisation results in a one time loss of ~150.000€ multiplied by long term loss in analytical power QMSS2 Peter Mohler 2010
Last Prolegomon Scientific Engineering • Scientific work requires transparent, inter-subjective methods • Not all methods are fully based on scientific proof • Some of them are based on experience • they “work”, but are not completely understood why • Doing surveys is scientific engineering • Improving survey research methods requires better scientific understanding why something works or works not (scientific laws of causality) QMSS2 Peter Mohler 2010
Designing for Reality QMSS2 Peter Mohler 2010
Doing Data Collection • Is like driving 100 Quadrigae simultaneously • Is a complex logistic enterprise • Needs lots of engineering QMSS2 Peter Mohler 2010
What is the Object of our Desire? QMSS2 Peter Mohler 2010
The Object of Desire QMSS2 Peter Mohler 2010
The Obscure Object of Desire • Is that single measurement (system – see Bo) • Everything else is environment (in systems terminology to that measurement system) • The “environment” is the big machine that supports the single measurement (asking a question and getting a response • Questionnaires are interconnected chains of single measurements (that is a system in itself) QMSS2 Peter Mohler 2010
The Environment of the Object of Desire QMSS2 Peter Mohler 2010
Target Measurement in a Nano Dimension QMSS2 Peter Mohler 2010
And its Environment (looks like data collection QMSS2 Peter Mohler 2010
Designing for Reality • Identify your research question • Identify your analytical needs • Apply parsimony • Observe cumulation QMSS2 Peter Mohler 2010
Define Your Research Question QMSS2 Peter Mohler 2010
See a Survey as a Production Process • “Survey Life Cycle” as a metaphor • CCSG build on that metaphor QMSS2 Peter Mohler 2010
Survey Life Cycle QMSS2 Peter Mohler 2010
Survey Production Process II • Introduce Analysis as a core production process item • Combine several items from the life-cycle QMSS2 Peter Mohler 2010
Production System QMSS2 Peter Mohler 2010
Identify Your Analytical Needs • Be very specific about the precision you really need • Be even more specific about the precision you really can achieve • Be suspicious about the volatility of your analytical devices (ask Paul) QMSS2 Peter Mohler 2010
Precision? How Fine is our Measurement Graded? QMSS2 Peter Mohler 2010
Apply Parsimony • AGE plus what other demographics? • 20%-30% of social surveys are about demographics • Experimental pre-research identifying strong predictor variables • Of the shelf surveys or special study ) QMSS2 Peter Mohler 2010
Observe Cumulation • Re-analyse, re-analyse, re-analyse • Drop items that were filtered out in re-analysis • Provide documentation that allows for cumulation QMSS2 Peter Mohler 2010
Reducing Complexity • see Bo’s presentation QMSS2 Peter Mohler 2010
Enjoy your survey! QMSS2 Peter Mohler 2010