Testing and Documentation Part I

1. Testing and Documentation Part I

2. Testing the questionnaire If the questionnaire doesn’t work at all or as intended: • considerable time and money will have been wasted • it makes the agency look bad • future funding may be unobtainable • it may be more difficult to get cooperation in the future • public trust may be lost

3. Testing can tell you… if others have the same definition and ideas about a concept as you do. • The same word may mean different things to different people. • It is worth investigating whether respondents have preconceived ideas about terms you use. • You may need to choose a different term or include wording in your questions to clarify your intent. • Interviewers may need to be trained to recognize and solve misunderstandings.

4. Testing can tell you… if a topic or question is sensitive or offensive. • Respondents may not be comfortable talking about some topics. • Examples may include earnings, paternity, religion, and disability. • Respondents may refuse to answer sensitive questions, may lie, or may quit the interview. These outcomes should be avoided. • Careful wording of questions may make them more acceptable. • Placing sensitive topics toward the end of the survey, when possible, minimizes data loss if a respondent ends the interview.

5. Testing can tell you… if everyone understands the question in the same way. • If people understand the question to mean different things, they will essentially be answering different questions. • They will not be giving answers that are comparable. • The resulting data may clearly not make sense. In this case, the results will not be used and analysis in that area will be absent. • The resulting data may not be obviously wrong. In this case, they may be used without knowing there is a problem but the resulting analysis will be meaningless and incorrect.

6. Testing can tell you… if the question was understood as intended. • Perhaps people understand the question to mean the same thing, but it is different than what the interviewer was trying to ask. • This may happen when the same words are used differently in common conversation and in the survey world. • The resulting data would likely not show signs of a problem and would be used. • The data would be consistent across respondents, but analysis would be incorrect as the data would not mean what the question indicates they should.

7. Testing can tell you… if interviewers are able to read the question as written. • For consistency across interviews, it is important that all respondents are asked the same questions. • If a question is tricky to pronounce or awkward to read, interviewers will tend to alter it. Not all interviewers will alter it in the same way however, introducing inconsistency.

8. Testing can tell you… if respondents are able to understand the question as read. • Questions may be too long, too complicated, have too many parts, or contain words the person does not know. • Respondents may give answers that don’t make sense, make up answers to avoid embarrassment, ask questions to clarify what was asked, or not answer. • Satisficing by respondents leads to nonsense data. • Respondents asking questions for clarification may produce good data, but it extends the length of the interview. • Respondents are usually more likely to supply an answer than seek to clarify the question when they don’t understand. • If respondents to not understand several questions, they may become uncomfortable and choose to end the survey.

9. Testing can tell you… if the answer categories provided are appropriate. • Responses may cluster in a category if it contains a trigger word. • Additional categories that should be included may be found. • Requests for clarification may indicate that the categories are not understood or that the boundaries between them are not clear. • Responses clustering at either end of a numerical range may indicate that it needs to be extended. • Confusion on hearing a numerical range may indicate that the range is surprisingly high/low to the person and they are reevaluating their understanding of the question.

10. Testing can tell you… if respondents are able to recall the desired information. • If the recall period is long, respondents may not clearly remember the needed details or how often the event happened. • If an event isn’t particularly memorable, the details or frequency of occurrence may have been forgotten.

11. Testing can tell you… if the skip pattern is correct. • Broken skips can take respondents to questions that do not make sense for them. • They may make up answers to the questions, resulting in bad data. • They may quit the interview in frustration. • This leads to data missing in sections where it should exist and bad data existing in sections that shouldn’t have been asked. if interviewers can follow the skip pattern. • Even if the skips are correctly written, they may be too complicated for interviewers to follow correctly. • When interviewers cannot follow the skips, the same problems result as come from broken skips.

12. Testing can tell you… if translations of the survey into other languages work well. • Construct equivalence is more important than linguistic equivalence. • A word for word translation might not have the same meaning in the new language as in the original. • Focus groups with speakers of the target language are desirable to determine if they think about any of the concepts differently. This can help choose between different possible terms to use in the translation. • Cognitive testing is needed to ensure that people get the same meaning from the questions and answer them in the same way.