Testing and Documentation Part II

1. Testing and Documentation Part II

2. Types of testing Focus groups • Gather several people to have a guided discussion • Particularly useful before developing questions • Give insight into how people think about issues and terms • May indicate what wording would be best understood • May reveal if there is sensitivity around certain issues or not • May identify differences by class, education, or other characteristics

3. Types of testing Vignettes • Participants are given hypothetical scenarios and asked to answer questions about them. • Especially useful when finding enough subjects with the desired traits would be difficult • Removes desire to give desirable responses • Focuses attention on particular questions, thought processes, or answer categories

4. Types of testing Cognitive interviews • Are used during questionnaire development • Multiple rounds are often done to test changes made from previous feedback • A and B versions may be tested simultaneously • Subjects can be asked to think out loud while answering the survey • Lets you understand their thought process • Clarifies how they are interpreting the question • May reveal issues with recall, saliency, or ability to calculate • Subjects can be probed about the survey questions to target issues the investigator is interested in • Subjects can be given vignettes rather than or in addition to, responding based on their experience

5. Types of testing Behavior coding • Observe interviews and take note of respondent and/or interviewer behaviors such as: • Pauses that indicate confusion, lack or understanding, or recall difficulty • Respondents asking for clarification • Respondents giving inappropriate or inconsistent answers • Interviewers rephrasing questions • Negotiations on which answer category is best when none really fit • When done testing content, can ask follow-up questions on issues that were noted • Can also be used when survey is in the field

6. Types of testing Interviewer debriefing • Done individually or in a group • Interviewers could answer written questions about the experience and note any problems, suggestions, and questions if not available in person • Can ask about: • Flow of questionnaire • Questions that are awkward to read or difficult to pronounce • Places where respondents needed help • Answers that were difficult to code or for which a suitable category did not seem to exist • Concepts, terms, and reference periods that need to better defined

7. Types of testing Expert reviews • Survey is given to people who are experts in survey design and/or the subject matter • Experts point out potential problems based on their experience and knowledge • Experts suggest question wording, answer categories, etc that may work better • Does not replace need to test questionnaire on respondents

8. Revising the questionnaire After a round of testing, revisions are made to try to fix identified problems. Additional testing is needed to see if the fixes work and/or if they cause new problems • Subsequent rounds should focus on the changes but also make sure the change didn’t negatively impact understanding of other questions and that no errors were accidentally introduced Multiple rounds of testing will be needed, but it can’t go on indefinitely. Good version control is very important. Document the test results, what changes were made and why, and results of retesting. • This allows for justification of decisions and contributes to knowledge. Avoids reinventing the wheel.

9. Tips and techniques for testing Indicate the version in the file name and also on the print version of the document. Not all types of testing need to be done, but it is good to do more than one type. Each reveals different kinds of issues. Testing should be done using the same mode as the actual interviews. If the questions will be read by an interviewer, don’t test them by having the respondent fill a paper questionnaire and vice versa. Different does not necessarily mean better. Changes need to be tested too.

10. Importance of documentation Details what is included in the survey and why • Explains concepts and how to make them • International standards and needed additions/variations to them • Shows possible paths through the survey • Results of testing Lists changes to the survey over time • When, why, impact, comparability • Necessary for data users making historical comparisons Furthers openness and transparency • Increases public confidence • Makes avoiding political pressure easier • Avoids having to repeatedly answer the same questions • Critics might not agree with method but can trust change in same series over time

11. What should be documented? Most everything • If you are thinking about including it in the documentation, it is probably worth having. • It is much easier to document something while it is fresh in the mind than having to remember the details later. Old documentation should be added to, not deleted and replaced. • All documentation should clearly marked as to what survey and time period it refers to. • Consistent titles should be given to the same type of document produced for different years.

12. What should be documented? Definite items to be made publically available and internally archived: • Public use data set • Publications and tables • Questionnaire (including skips) • Concept dictionary with verbal definitions and instructions on variable edits • Data dictionary including variables and their locations on the data file • Interviewer training materials • Changes to the survey over time • Response rates, sample design, weighting methods, and survey metadata • Corrections to publications and released data • Contact information for data users with questions

13. What should be documented? Definite items to have for internal use: • Original and edited data sets with personally identifying information that cannot be shared with the public • Data cleaning techniques and data edits stored on the file • Significance tables/confidence intervals • Procedures for going from raw data to publication • Where data files are stored • Results of testing Materials made for internal use can generally be shared with data users except when it would violate respondent confidentiality

14. Examples of documentation • Uganda Concept Dictionary • Uganda Interviewer Training Manual • Uganda Survey Flowchart