Internal Experimental Validity: Single System Designs

2. Internal Experimental Validity: Single System Designs • History. Did any other events occur during the time of the practitioner-client contacts that may be responsible for the particular outcome? Since single-system designs by definition extend over time, there is ample opportunity for such extraneous events to occur. Example: During the course of marital therapy, Andrew loses his job while Ann obtains a better position than she had before. Such events, though independent of the therapy, may have as much or more to do with the outcome than did the therapy. • Maturation. Did any psychological or physiological change occur within the client that might have affected the outcome during the time he or she was in a treatment situation? Example: Bobby was in a treatment program from the ages of 13 to 16. But when the program ended, it was impossible to tell if Bobby had simply matured as a teenager or if the program was effective. • Testing. Taking a test or filling out a questionnaire the first time may sensitize the client so that subsequent scores are influenced. Example: The staff of the agency filled out the evaluation questionnaire the first week, but most staff members couldn’t help remembering what they said the first time so subsequent evaluations were biased by the first set of responses. • Instrumentation. Do changes in the measurement devices themselves or changes in the observers or in the way the measurement devices are used have an impact on the outcome? Example: Phyllis, a public health nurse, was involved in evaluating a home health program in which she interviewed elderly clients every 4 months for a 3-year period. About halfway through, Phyllis made plans to be married and was preoccupied with this coming event. She thought she had memorized the questionnaire, but in fact she forgot to ask blocks of questions, and coders of the data complained to her that they couldn’t always make out which response she had circled on the sheets.

3. Continued • Dropout. Are the results of an intervention program distorted because clients drop out of treatment perhaps due to anxiety, low motivation, and so on, leaving a different sample of persons being measured? Example: While studying the impact of relocation on nursing home patients, a researcher noticed that some patients were refusing to continue to participate in the research interview because they said that the questions made them very anxious and upset. This left the overall results dependent on only the remaining patients. • Statistical regression. Statistically, any extreme score on an original test will tend to become less extreme upon retesting. Example: Hank was tested in the vocational rehabilitation office and received one of the lowest scores ever recorded there, a fact that attracted some attention from the counselors who worked with him. However, because of a bus strike, Hank couldn’t get to the center very often, and yet upon retesting, he scored much better than he had before. Had Hank been receiving counseling during that period, it might have appeared as though the counseling produced the changes in score. • Diffusion or imitation of intervention. When intervention involves some type of informational program and when clients from different interventions or different practitioners communicate with each other about the program, clients may learn about information intended for others. Example: Harvey was seeing a practitioner for help with his depression at the same time as his wife Jean was seeing a practitioner for help with anger management. Although their intervention programs were different, their discussions with each other led them each to try aspects of each other’s program.

4. External Experimental Validity: Single System Designs • Interaction between the intervention and other variables. Any intervention applied to new clients, in new settings, by new practitioners will be different from the original intervention in varying degrees, thus reducing the likelihood of identical results. Example: Esther’s friend Louise was much taken with her tales of using self-monitoring with adolescent outpatients, so she tried to adapt the technique to youngsters in her school who were having problems related to hyperactivity. Even though she followed Esther’s suggestions as closely as possible, the results did not come out as well as they did in the other setting. • Practitioner effect. The practitioner’s style of practice influences outcomes, so different practitioners applying the same intervention may have different effects to some degree. Example: Arnold had been substitute teaching on many occasions over several years in Marie’s class and knew her course materials quite well. However, when he had to take over her class on a permanent basis because she resigned with a serious illness, he was surprised that he wasn’t as effective in teaching her class as he had expected. • Different dependent (target or outcome) variables. There may be differences in how the target variables are conceptualized and operationalized in different studies, thus reducing the likelihood of identical results. Example: Ernesto’s community action group defined “neighborhood solidarity” in terms of its effectiveness in obtaining municipal services, defined as trash removal, rat control, and the expansion of bus service during early morning hours. While Juan used the same term, neighborhood solidarity, with a similar ethnic group, the meaning of the term had to do with the neighbors presenting a united front in the face of the school board’s arbitrary removal of a locally popular principal.

5. Continued • Interaction of history and intervention. If extraneous events occur concurrently during the intervention, they may not be present at other times, thus reducing generalizability of the results of the original study. Example: The first community mental health center opened amid the shock of recent suicides of several young unemployed men. It was forced to respond to the needs of persons facing socioeconomic pressures. However, when a new center opened in a nearby area, there was no similar population recognition of this type of counseling, and considerably fewer persons went to the second center than to the first. • Measurement Differences. To the extent that differences exist between two evaluations regarding how the same process and outcome variables are measured, so too are the results likely to be different. Example: The standardized scales used by Lee with his majority group clients were viewed by the several minority group members as being too offensive and asking questions that were too personal, so he had to devise ways of obtaining the same information in less obvious or offensive ways. The results continued to show considerable differences between majority and minority clients. • Differences in Clients.The characteristics of the client-age, sex, ethnicity, social class, and so on-strongly affect the extent to which results can be generalized. Example: Nick devised a counseling program that was very successful in decreasing dropouts in one school with mainly middle-class students. But when Nick tried to implement his program in a different school, one with largely low-income students, he found the dropout rate was hardly affected.

6. Continued • Interaction between testing and intervention.The effects of the early testing-for example, the baseline observations-could sensitize the client to the intervention so that generalization could be expected to occur only when subsequent clients are given the same test prior to intervention. Example: All members of an assertion training group met for 3 weeks and role-played several situations in which they were not being assertive. The formal intervention of assertion training started the fourth week but by then all group members were highly sensitized to what to expect. • Reactive effects to evaluation. Simple awareness by clients that they are in a study could lead to changes in client performance. Example: All employees at a public agency providing services to the elderly were participants in a study comparing their work with work performed by practitioners in a private agency. Knowing they were in the study, and knowing they had to do their best the public agency practitioners put in extra time, clouding the issue as to whether their typical intervention efforts would produce similar results.

7. Best Design toEliminate Alternative Explanations Solomon Four Group Design