1 / 30

Task Force on Improvement of Response Rates and Minimisation of Respondent Load

Per aggiungere alla diapositiva il logo della società: Scegliere Immagine dal menu Inserisci Individuare il file con il logo della società Scegliere OK Per ridimensionare il logo:

cally-willis
Download Presentation

Task Force on Improvement of Response Rates and Minimisation of Respondent Load

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Per aggiungere alla diapositiva il logo della società: • Scegliere Immagine dal menu Inserisci • Individuare il file con il logo della società • Scegliere OK Per ridimensionare il logo: • Fare clic su un punto qualsiasi del logo. Attorno all'oggetto verranno visualizzati i quadratini di ridimensionamento. • Per ridimensionare l'oggetto, utilizzare i quadratini. • Per conservare le proporzioni dell'oggetto da ridimensionare, trascinare i quadratini di ridimensionamento tenendo premuto MAIUSC. JOINT EUROPEAN COMMISSION – OECD WORKSHOP ON INTERNATIONAL DEVELOPMENT OF BUSINESS AND CONSUMER TENDENCY SURVEY Task Force on Improvement of Response Rates and Minimisation of Respondent Load Part 1 – Relationship between response rates and data collection methods Bianca Maria Martelli, Institute for Studies and Economic Analysis -ISAE – Rome, Italy BRUSSELS, 14-15 November 2005

  2. TERMS OF REFERENCE 1. DATA COLLECTION METHODS which is the most effective for: • Improving response rates • Reducing burden • Impact on costs • Sectoral effects • Size effects 2. COMMUNICATION METHODS most effective and cost efficient for: • Compulsory surveys • Voluntary participation Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  3. ACKNOLEDGEMENTS The present research has been realised with the substantial contributions of the members of the EC/OECD task force on RR who supported me with extremely useful material , references, information, personal experience and suggestions. Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  4. WHY IS THE RESPONSE RATE IMPORTANT? • Response Rate (RR) one of the most important factors to ensure a high quality of the BTS results (Etter, 2002) • Non response rate (NR) is one of the components of non-sampling error that may affect the survey and the most efficient strategy for reducing measurement errors is to eliminate possible causes of such errors during the whole survey design stage (OECD 2003) Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  5. WHY FIRMS/CONSUMER RESPOND? Conceptual framework for COS/BTS Survey participation (proposed by Willimack 2002, reported by Petroni 2004): • Businesses weight response burden against business goals when considering the decision to participate in a survey. Both burden and goals may be impacted by: • factors relating to the external environment, the business and the respondent, which are not under control of the survey organisation • survey design features, which are under the survey organisation control Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  6. FACTORS INFLUENCING RESPONSE RATES Kind of Survey Subject Reporting, Sampling, Response Unit Target Universe RR Questionnaire Frame Sampling design Data Collection Mode Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  7. WHAT IS (NON) RESPONSE RATE? • CONFUSION ON THE TOPIC : people deals with RR/NR, but not explain what really apply • NEED OF CLEAR RR/NR DEFINITION The response rate is the number of complete/incomplete interviews with reporting units divided by the number of eligible reporting units in the sample (AAPOR 2004) • SEVERAL KIND OF RR/NR • UNIT / ITEM NR Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  8. KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003) (1/3): where n’ = number of enterprises which did not submit useable information n = number of enterprises in the survey Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  9. KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003)(2/3): In case of unequal sampling fraction but uniform weights a proper measure is the following: where is the sampling fraction of the ith unit Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  10. KINDS OF NON RESPONSE RATES OECD DEFINITIONS (2003) (3/3): In the most general case, of unequal sampling fraction and reporting units with different weights: where wi is the size weight of the ith unit Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  11. KINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS (AAPOR 2004) • COOP= Co operation rate Proportion of all cases interviewed of all eligible units ever contacted • REF = Refusal Rate Proportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases • CON = Contact Rate Proportion of all cases in which a respondent refuses to do an interview, or breaks-off an interview of all potentially eligible cases Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  12. KINDS OF (NON) RESPONSE RATES OTHER LITERATURE ON RR/NR DEFINITIONS • OVERALL COMPLETION RATE(Lozar Manfreda, 2003) % of responses among all sent invitations or exposed to invitation/eligible, including partial and complete respondents The author assumes that most often researchers when dealing with RR actually refer to it (similar to OECD NR1) • FULL COMPLETION RATE Considers only the % of complete (unit) respondents • DROP-OUT RATE Referring only to item non response • CLICK-OUT RATE For Web surveys (% of accessing the Web questionnaire among all invited) Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  13. RECOMMENDATION ON (NON) RESPONSE RATES • Clearly specify which kind of response rate is applied According to the specific work purposes of this paper: • OECDNR1 (or OVERALL COMPLETION RATE) is useful inthe first steps of the survey designto check the robustness of the data collection process. • FULL COMPLETION RATE , DROP OUT RATE, CLICK TROUGH are also useful for analysing the DCM Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  14. DATA COLLECTION MODES TECHNIQUES • FACE TO FACE • Historically one of the first applied (since 1912) • Mainly for COS • High quality, High costs • CAPI/CASI Computer Assisted Personal Intervieving/Computer Asssisted Self- Interviewing • “Computerised” version of face-to face • MAIL • The first historically applied (1788 as reported by De Leeuw, 2005) • Implies the self-administration of questionnaire • Visual approach • “Low” costs, relatively low RR Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  15. DATA COLLECTION MODES TECHNIQUES • FAX • Similar to MAIL • Widely used for BTS • TELEPHONE • Useful as supporting measure • “Aural” (instead of “visual”)contact: risk of “loss of context” • Primacy/recency effects (for COS) • Primary role of interviewer • CATI (Computer Aided Telephone Interview) • Widely used for COS, less for BTS (Italy, CB) • Allows to quickly collect information • Allows immediate control of inconsistencies • Expensive Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  16. minus: Costs (in comparison with postal tecnique) Possible subjective biases Focus: CATI AT ISAE for BTS Since 1988 for Business (since 2002 completely) Since the beginning (1992) for Services plus: • Rapidity • Reliability • High response rate • Less sample biases • Easy substitution Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  17. Focus: CATI AT ISAE for BTS Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  18. DATA COLLECTION MODES TECHNIQUES • E - MAIL • “Visual”approach, similar to MAIL and FAX • Decreasing trend • Low cost • Confidentiality problems (not allowed at INSEE ) • TOUCH-TONE Data Entry/VOICE RECOGNITION Entry • For COS • Low RR • Low cost Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  19. DATA COLLECTION MODES TECHNIQUES • WEB/CAWI • Increasing relevance for BTS (less for COS) • Low cost • Coverage errors • Risk of loss of context in the questionnaire • Higher Unit NR/Lower Item NR (vs MAIL, Annaud, 2005, Etter) • Self selection bias • Only a part of participants (30-50%) freely choose this mode (Etter, Stangl, Kershoff) Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  20. STEPS (timing of interaction) CO OPERATION / PRE SURVEY NOTIFICATION OVERALL RESPONSE STAGE FULL COMPLETION REMINDINGS FOLLOW UPs DATA COLLECTION MODES PROCESS DCM often is a MULTISTAGE PROCESS and for any step more than one mode can be applied MODES the different modes can be applied : - CONCURRENT and/or in a: - SEQUENTIAL way Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  21. MULTIMODE SYSTEM OPTIMIZE data collection procedures REDUCE total survey ERRORS within the available time and costs DATA COLLECTION MODE: MIXED MODE (De Leeuw, 2005) Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  22. FIRM REDUCES BURDEN REDUCES PANEL ATTRITION (TIME IN SAMPLE) INCREASES VOLUNTARY PARTICIPATION MIXED MODE DATA COLLECTION EFFECTS ON BURDEN, COSTS, RR INSTITUTION INCREASES RR INCREASESBURDEN INCREASES COSTS ERRORS DUE TO MODE EFFECTS Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  23. SECTOR Firms specific effect is present Differences (for both Mail and Internet) (Etter, 2002) Differences (for both CATI and FAX), Italy …..but limited Retail, Services lower RR (but also smaller size…) SIZE Generally inverse correlation between RR and Size medium/large firms respond more regularly (Etter, 2002) Burden is greater for small firms (Petroni, 2003) Some exception South Africa (Kershoff, 2005), Latvia do not notice remarkable influence SECTOR AND SIZE EFFECTS Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  24. COMMUNICATION METHODS (CO OPERATION STAGE) • Action useful in getting co operation (and reducing NR): • Tailoring contacts to the characteristics of the establishment • Contacting the “right” person (mid level management) • Overcoming the lack of awareness (of the Survey / Institution) • Building the initial relationship (personal contact advisable) • Knowledge of survey characteristics • Survey pre notice Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  25. COMMUNICATION METHODS: INTERVIEWERS (CO OPERATION STAGE) It is strongly advisable: • Use of interviewers • in the initial step, even if the ongoing DCM is different, increases RR (BLS Survey) • Training of interviewers • Mainly for BTS, less for COS • Overcome gate keepers: difficult first contact does not necessarily mean unwillingness to participate Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  26. COMMUNICATION METHODS (CO OPERATION STAGE) all the above mentioned actions are essential also for effectively communicating: • COMPULSION (OF RESPONSE) as it may increase the (perceived) burden of respondent (Mixed ) Effects of compulsions : • Increases the Overall Completion Rate (lowers the Unit NR) • May increase the Drop Outs (Item NR) • Positive effects in France, mainly for large firms • Positive effects in Italy (Retail) Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  27. COMMUNICATION METHODS (CO OPERATION STAGE) • Voluntary participation most effective approaches : • Sending the business survey to named individuals and use telephone follow-up methods to encourage responses (Paxon, 1995) • Lowering the response burden (Linn 2003) • Increasing the perceived relevance of the survey to business goals (Linn, 2003) • Adopting a Mixed DCM Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  28. OVERALL RESPONSE STAGE : PANEL ATTRITION • Maintain contacts with participant • Renewing interest • Using simple and plain questionnaires • Avoiding open ended and “difficult to answer questions” • Question wording • accuracy, mainly when using concurrent DCMs • Remindings • Follow – ups Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  29. CONCLUSIONS AND RECOMMENDATIONS • Efforts should be made in improving preliminary contacts and remindings / follow up actions • All the modes have plusses and drawbacks • The starting point of view should be the firm (to a lesser extent the consumer): adopting a company-centric point of view • Efforts should be made to allow surveys’ participants to choose the mode they prefer Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

  30. CONCLUSIONS AND RECOMMENDATIONS • Although Internet (online) seems to be the mode of the future, a large part of firms still prefer the most traditional mail questionnaire and a significant part of respondents prefer fax (habit persistence) • The future appears to be even more bound to a mixed data collection mode • Mixed mode data collection should be carried out both concurrently and in a multi stage way (as far as possible bound to BTS and COS timeliness) Joint OECD/European Commission Workshop on International Development in Business and Consumer Surveys 14-15 November 2005

More Related