Session 20: Data Collection in the Field

Session 20: Data Collection in the Field Ina F. Wallace RTI International IES/NCER Summer Research Training Institute, June 2007

Data Collection in the Field Topics Covered • Methods of Data Collection • Use of State-of-the-Art Data Collection Procedures • Ensuring Data Quality • Conducting Small-Scale Validation Studies • Ethical Issues in Data Collection

Methods of Data Collection • Interviewer Administered • In Person • Surveys • Assessments • Telephone • Self-Administered • Paper and Pencil • Mail-in • In-person • Computer Assisted Self Interview • Web procedures

Data Collection Mode Characteristics Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality

Advantages Maximum degree of communication and interaction Can utilize PAPI or CAI Allows for long, complex interviews Variety of tactics can be used to gain cooperation Response rates usually high Disadvantages High cost due to travel Can generate social desirability for sensitive questions Interviewers can affect respondents Can generate interviewer variance Presence of others can affect responses Falsification of interview Face-to-Face Interviewing

Advantages Cost lower than face to face More easy to provide training, supervision, and feedback Social desirability bias less than with face to face Interviewer variance less Disadvantages Less flexible No ability to use visual aids Must be shorter Questions must be less complicated Less capacity for motivating the respondent No coverage of non-telephone units Response rates lower Telephone Interviewing

Advantages Lowest cost Low social desirability Can utilize visual aids Allows respondent to take time and look up information (if necessary) No interviewer variance Disadvantages Little control over who completes the survey Item non-response high Response rates lower Require a long field period to obtain an acceptable response rate Respondents must have good literacy skills Limit to closed-format questions Mail Surveys

Computerized Procedures • Computer Assisted Personal Interview (CAPI) • Computer Assisted Telephone Interview (CATI) • Computer Assisted Self Interview (CASI) • Audio Assisted Self Interview (ACASI) • Telephone Audio Assisted Self Interview (T-ACASI) • Computer Assisted Data Entry (CADE) • Web Surveys

Advantages Complex skip patterns and branching are easier to follow Previously acquired information can be programmed Ranges, inconsistencies, and illegitimate values can be checked Help screens can be included Data are available for analysis more quickly Disadvantages Time consuming to develop and program computer Must be adequately tested Can’t change things easily once fielded Risk of computer crashing Costly Computer Assisted Procedures

Direct Assessment • Used to obtain psychological and educational information about an individual • Individually administered • Research driven NOT for clinical or diagnostic purposes • Follows a standardized protocol

Observational Techniques • Involve a researcher making observations within a natural setting • Direct Observation - individuals know they are being observed • Continuous Monitoring recording as much behavior as possible • Time Allocation randomly visiting the sample to observe at different times • Unobtrusive Observation – individuals do not know they are being observed

Advantages No or limited participation needed May be good for generating hypotheses Sensitive social issues can be obtained more accurately High in validity because of the depth of information collected Disadvantages Variance associated with observers is similar to that found with interviewers Can be very costly as they are often labor intensive Time consuming Reliability can be low Generalizability can be low Observational Techniques

Summary: Issues in Choosing a Strategy • Type of Population • Nature of Research Issue • Question Form • Question Content • Response Rates • Costs • Length of Data Collection

Methods of Administration • Standardized Interviewing Approach • A protocol in which interviewers interact with the respondent in a manner which is consistent across all interviewers • Conversational Interviewing • A protocol in which interviewers modify and adapt questions to the respondent’s situation • Hybrid Interviewing • Combination of thetwo approaches

Factors that Can Explain Interviewer Variance • Respondent • Characteristics • Knowledge • Interest/motivation • Confidence • Strength of convictions • Expectations • Interviewer • Characteristics • Appearance • Motives • Beliefs/attitudes • Perceptions • Expectations • Skills/Knowledge Interviewer Effects • Survey Conditions and Setting • Mode of interview • Standardization • Interviewer training • Interviewer supervision • Monitoring/observation • Questionnaire • Definition clarity • Terminology/jargon • Question form • Instructions • Question wording • Question topic From Biemer & Lyberg (2003). Introduction to Survey Quality

Role of the Interviewer/Assessor • Gaining Cooperation • Converting reluctant participants • Recognizing and dealing with refusals • Training and Motivating Respondents • Interviewers implicitly and explicitly teach respondents how to behave • Obtaining Quality Data • Ensuring Confidentiality

Assessing Children • Assessor Characteristics • Flexibility • Vigilance • Self-awareness • Eye contact • Establishing Rapport • Allaying apprehensions • Helping children feel at ease • Maintaining limits on permissible behavior • Arranging the Physical Environment

Ensuring Quality Data • Recruiting interviewers/assessors • Characteristics of all data collectors • Characteristics of interviewers • Characteristics of assessors • Training on standardized procedures • Certification as data collectors • Monitoring performance

Content of Training Presenting the study Contacting participants Administering the questionnaire/ assessment Probing Recording responses Confidentiality Interpersonal relations Administrative procedures Procedures for Training Written materials Lecture and presentation Computer-based tutorials Planned exercises Role Playing Observation Certification Training

Interviewer Training Agenda • Introduction to the Study • Role of the Interviewer • Confidentiality Procedures • Review of Questionnaire and Protocols • Standardized Interviewing Techniques • Round Robin Interview • Gaining Cooperation • Presentation of Scenarios • Round Robin Demonstrations • Paired Mocks • Certification

Assessor Training Agenda

What to Monitor Detection and prevention of falsified information Compliance with rules and guidelines about administering the questionnaire or assessment Performance on non-interview/assessment tasks How to Monitor Telephone verification with a sample of respondents Direct observation – in-person or by telephone Audio-recording – CARI Review of materials Performance and production measures Monitoring Performance

Human Subjects Issues • Institutional Review Boards • Informed Consent • Who is carrying out research under whose sponsorship • Description of purposes and procedures • Duration of involvement • Risks and Benefits • Confidentiality • Voluntary nature of research including stopping early and skipping questions • Compensation • Contact information • Confidentiality Issues

Data Collection Evaluation Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality

Post-Data Collection Evaluations • Experiments • Observational Studies • Internal Consistency Studies • External Validation Studies • Administrative Record Check Studies • Re-interview/Re-Test Studies

Session 20: Data Collection in the Field