RTI International is a trade name of Research Triangle Institute

Considerations When Using Family Surveys with Spanish-Speaking Families Georgina McAvinchey Murrey Olmsted, Ph.D. Don Bailey, Ph.D. Measuring Child and Family Outcomes National TA Meeting Baltimore, MD August 27, 2007 RTI International is a trade name of Research Triangle Institute

Goals for this Session • Provide background of issues and early findings • Describe cognitive testing as a methodology for understanding how respondents understand and respond to survey directions and items. • Present findings from a study using cognitive testing interviews with 40 Spanish speaking families in early intervention programs • Initiate a forum for discussing ways to maximize valid participation in state surveys by families whose primary language is not English

Background Information • Families participating in early intervention programs are ethnically diverse Percent of Participants

Background Information • The number and proportion of Hispanic/Latino families enrolled in early intervention is increasing Number of Hispanic/Latino Families Enrolled

Background Information • Hispanic/Latino families enrollment in early intervention is growing much faster than African American enrollment

Background Information • The NEILS study found differences by race/ethnicity in initial experiences with early intervention (Bailey et al., 2004)

Background Information • A more recent NEILS paper (Bailey et al., 2005) showed that race/ethnicity was a significant predictor of family outcomes at 36 months of age • White families significantly higher than black • Black families significantly higher than other minority groups • A forthcoming NEILS paper (Bailey et al., in press) using structural equations modeling found that white families tended to report more positive perceptions of quality of family services and were more likely to report that EI had a positive effect on their family

Background Information • A recent pilot study of the Family Outcomes Survey showed substantial differences in outcomes reported by English-speaking versus Spanish-speaking families

Background Information

Study Goals • Fundamental Question • Do differences in responses represent true differences in outcome attainment or might they reflect problems with translation, instrument format, different cultural interpretations of the meanings of items, or different expectations for what programs can and should do for families? • Research Sponsor • This project was sponsored by the Early Childhood Intervention Division (ECID) of the Texas Department of Assistive & Rehabilitative Services • Research Approach • To answer this question, we conducted a set of cognitive testing interviews with 40 Spanish-speaking families in Illinois and Texas

What is Cognitive Interviewing? • Cognitive interviewing is a qualitative testing methodology used to explore common problems in the way information is displayed, questions are asked, and tasks are presented to respondents in questionnaires (Willis, Royston, Bercini, 1991, Willis, 1994, 2005). • Based on a cognitive psychology model of the mental processes involved in responding to questionnaires and surveys by Tourangeau (1984) and Eisenhower et al. (1991) • Encoding in memory (have some knowledge/memory of the question) • Comprehension (understand the question and relevant concepts) • Retrieval (retrieve the information from memory) • Judgment (assess the completeness/relevance of their memories) • Communication (decide whether their answer fits the answer categories provided and also decide whether they actually want to provide an answer or provide one that might be socially acceptable)

What Kinds of Questions Does it Address? • Using structured interview techniques, a methodologist examines the following kinds of questions: • What do respondents think the question is asking? • What do specific words and phrases in the questions mean to respondents? • Are words used that respondents do not understand? • Are the memory and/or decision requirements of the questionnaire reasonable? • How do respondents choose their answers? • How well do respondents understand instructions and visual cues?

How Does it Work in Practice? • Identify issues in a questionnaire that may be problematic. This can often be done via a structured instrument review process. • Develop a cognitive testing guide to address each issue • Recruit participants that are representative of the target audience • 8-12 participants are typical for an average questionnaire • If federally funded, OMB restricts to 9 interviews per questionnaire • Conduct testing interviews in a one-to-one setting • Select a quiet location for the interviews where you will not be disturbed • Record all interviews and take good hand-written notes • Provide an incentive as a “Thank you” gift to participants • Review interviews and summarize the findings and the recommendations for possible changes to the questionnaire • Discuss the findings and implications with the project team and determine which changes are appropriate

Probing Techniques Used in Interviewing • Primary approaches for probing used in cognitive interview are concurrent and retrospective probing. • Concurrent probing: This involves asking the participant questions as they fill out the questionnaire to explore their understanding of concepts, questions, and other related factors. • Retrospective probing: Participants are first asked to complete a copy of the questionnaire and then are asked questions about their understanding of terms, questions, and concepts as the interviewer reviews the questionnaire with them. • Some examples of types of probes used include: • Comprehension probe: What do you think is meant in the question by “your child’s development? • Paraphrase probe: In your own words, what is this question asking? • Confidence judgment probe: Do you feel that you know enough about your “child’s development? Explain. • Elaborative/Expansive probe: Can you explain? • Recall probe: Think of the person or persons that provide services to your child. Do you feel that you can adequately communicate with them? Explain.

What Are the Results of Cognitive Interviewing? • Develop a cognitive testing summary report providing an overview of the background, methods, findings, and recommendations • Identification of problematic questions • Recommendations for possible solutions • Marked-up questionnaire showing possible changes • Vet possible changes to ensure that solutions address findings and do not cause further problems • Ensure that the meaning of the questions are not lost in recommended changes • Evaluate the language and cultural implications of any changes • Update and test the new version of the questionnaire

Background of the Spanish Testing • ECID requested that RTI conduct cognitive testing of the FOS with Spanish speaking families in Texas and Illinois • Earlier pilot data of the original Spanish version of the FOS showed that families had: • Lower average scores on many items • Lower overall response rates • Significant item-level missing data • Testing sought to determine whether predominantly Spanish-speaking parents are able to understand and respond to the survey easily

Spanish Cognitive Interview Protocol • Protocol was based on the materials that were used to test the English version of the FOS during the summer of 2006 • Also included new questions about access to services and experience with early intervention • Previous research support the idea that cognitive testing interviews can provide valuable insights when testing cross-cultural survey instruments • It can reveal ways in which cultural differences may affect how respondents perform cognitive tasks involved in survey responding

Methodology and Participants Cognitive testing were completed with: • 40 family members • 6 sites across Texas and Illinois • Qualified participants were Hispanic/Latino families in Texas and Illinois • Interviews conducted in June and July 2007 • 90 to 120 minute interviews • $70 cash incentive

Recruiting Participants • Recruiting methods: • Posting flyers at early intervention centers • Utilized contacts established by coordinators, family contact personnel and parent liaisons to reach out to families • Focused recruiting effort on obtaining: • Parents or family members who had some caretaking role of a child who was currently enrolled in an early intervention program • Spanish speakers who are monolingual or bilingual with Spanish language dominance

Methodology: Inclusion Criteria • Inclusion criteria: • Linguistic skills (monolingual and bilingual speakers with Spanish dominance) • Education • Region of origin • Demographic diversity

Language Dominance • Language dominance was assessed for each participant: • Evaluated the language that the participant spoke in the home and studied in early school • Current usage of language • Language the participant feels most comfortable using for communication • Participants were classified as: • Monolingual English speaker (speaks only English; not eligible) • Monolingual Spanish speaker (speaks only Spanish, eligible) • Bilingual, English dominant speaker (speaks both Spanish and English; dominant in English; not eligible) • Bilingual, Spanish dominant speaker (speaks both Spanish and English; dominant in Spanish; eligible)

Respondent Profile • Demographics - 40 family members in 6 sites across Texas and Illinois • Language dominance characteristics • 6 monolingual Spanish speakers • 34 bilingual speakers with Spanish dominance • 3 men and 37 women • Median age 32 (Min: 23, Max: 53) • Exposure to early childhood service ranged from 1 to 4 years • Typically referred to early intervention by hospitals and/or pediatricians, except for a few children whose parents requested a referral

Respondent Profile: Spanish-speaking respondents • Countries represented by participants included: Mexico Peru El Salvador Dominican Republic Guatemala Ecuador Honduras Venezuela • The majority of respondents (75%) reported being of Mexican descent

Findings and Recommendations • Presentation of the finding and recommendations will be presented as follows: • General Findings Related to Early Intervention Services • Specific Instrument Design Issues • Example of Findings and Recommendations • Overall Recommendations

General Findings Related to Early Intervention Services • Participant’s Experience and Understanding of Early Intervention • Most participants praised the early intervention services Tenía una venda y me la quitaron. Me hicieron ver las cosas y aprender. Fue una nueva experiencia.” (I was blindfolded and they took it off. They made me see and learn things. It was a new experience.)

General Findings Related to Early Intervention Services • Desire to have early intervention services expanded past the 3 years of age • Parents’ initial expectationsoriginated from first Early Intervention meeting(s) • Parents expectations have been met and exceeded in several cases • Expected some type of improvement in their child’s special situation • Eager to learn how to better interact and help their children

Impact of Language on Early Intervention Services • Good communication with service providers • Service providers responded very well to the child’s needs • Some language barrier “Si yo pudiera hablar inglés estaríamos mejor, nos entenderíamos mucho mejor y me entenderían más lo que yo les trato de decir” (It would be better if I could speak English, we could understand each other much better, and they would understand what I try to tell them). • Fairly high overall satisfaction with having bilingual personnel available

Specific Instrument Design Issues • Participants generally liked the FOS questionnaire • A few consistent issues emerged from the cognitive testing which should be addressed including: • Instructions and 7-point Scale • Problems with Concepts and Terminology • Issues Affecting Comprehension of Survey Items and Instruction

Instructions for answer option 2, 4, and 6 • If a statement almost describes your family, but not quite, circle the number just to the left or the right. For example if you feel that the statement 5 “We know a good amount about dinosaurs” almost describes your family, but not quite—circle the 4. How much does your family know about dinosaurs?

7-point Scale • Many participants appeared uncomfortable or struggled using the scale during the first few questions • The spaces between response options were described as confusing by some but not all participants. “Estos huequitos de en medio confunden.” (These little holes in the middle are confusing.)

Instructions and 4-point Scale • Recommendations for possible change to the FOS • Eliminate instructions for answer option 2, 4 and 6 • Remove spaces between response options (see below)

Problems with Concepts and Terminology • Most of the concepts found in the questionnaire were well understood by parents • Some concepts and terms were confusing or needed clarification • Incorrect assumptions about participant’s knowledge of the terminology • Others were associated with vagueness • Some additional issues are presented in the question by question review of the report

Key concepts that caused difficulty • Child - Most participants understood this term as intended but there were a few parents with more than one child receiving early intervention (EI) services who were unsure which child to consider when completing the survey • Early Intervention Services (EI services) - The inclusion of this term in the survey assumed parental knowledge of early intervention services. Most participants reported lacking this knowledge prior to receiving these services for their children • Special needs - This term was found to be sensitive, and widely misunderstood by participants. Several of the participating family members associated this term with severe physical or mental impairments

Issues affecting Comprehension • Reading level – Many questions appeared to be over the participant’s current reading level and limited vocabulary. “Si se me hace difícil a veces el español. El español es diferente. A veces hay palabras que uno no las usa y si no las usa, a veces no está uno seguro si uno sabe lo que quiere decir. Esta muy elevado la forma de preguntar. Esta muy formal. Lo pondría en palabras más sencillas…” (Sometimes I have a hard time with Spanish. Spanish is different. There are some words that if you do not use them for a while, you are not sure what they mean. The way of asking is too “high”. It’s too formal. I would use simpler words…)

Issues affecting Comprehension • Unnecessary wording in some questions – Parents had trouble understanding some questions due to unnecessary or confusing text appearing in the questions Las preguntas no van directo al punto. Dan primero una… dan como dos preguntas en una sola pregunta. Simplificaría haciendo las preguntas más directas”. (Questions should go straight to the point. Sometimes they ask like two questions in one. I would simplify them by making them more direct).

Issues affecting Comprehension • Cultural issues – There were some cultural issues and assumptions that might have affected comprehension Creo que es tabú, porque si dices que tu niño es autista, creen que es ‘mongolito’, y como si fuera malo. Como aceptar que tu hijo es un perdedor. No solo en la comunidad mexicana, sino en los latinos. No sabemos del tema.” (I think its taboo, if you mention that your child is autistic, they think he is retarded, as if it was something bad. It is like admitting that your child is a loser. This not exclusive to Mexicans but the whole Latino community. (In our culture) we do not know about this.)

Example of Findings: Question 2 • Example: • Question 2 (Original): Algunos niños tienen necesidades médicas especiales, alguna discapacidad o retraso en el desarrollo. Estas a menudo se conocen como “necesidades especiales”. ¿Qué tan familiarizada está su familia con las necesidades especiales de su hijo? (Some children have special health needs, a disability, or are delayed in their development. These are often referred to as “special needs.” How familiar is your family with your child's special needs?)

Example of Changes: Question 2 • Example: • Question 2 (Revised): ¿Qué tan familiarizada está su familia con las necesidades especiales de su hijo? (Las necesidades especiales incluyen necesidades médicas, alguna discapacidad, problemas del habla, visión o audición, o demora en el desarrollo.) [How familiar is your family with your child's special needs?) (Special needs include health needs, a disability such as a speech, vision or hearing problem, or delayed development).]

Overall Recommendations • Simplify the survey items by moving from a two-sentence format to a single statement • Reduce the number of scale points from 7 to 4 by eliminating the current ratings of 2, 4, and 6 for which there are no behavioral anchors • Revise each question to clarify the meanings of key terms/concepts and simplify the provided response options • Make selected changes in the Spanish translation using words that are more likely to be understood by the majority of Spanish speakers

Questions & Answers • Any questions?

RTI International is a trade name of Research Triangle Institute