Was the program effective?

1. Was the program effective? Compared to what?

2. In most outcome studies DIFFERENCE is what is looked at!!!! • When looking at differences, the question to be answered is • COMPARED TO WHAT? • ANYTIME A QUESTION INVOLVES LOOKING FOR A DIFFERENCE, YOU MUST KNOW WHAT YOU ARE COMPARING. YOU CAN’T MEASURE A DIFFERENCE WITHOUT COMPARING!!! HERE ARE RESEARCH QUESTIONS THAT INVOLVE COMPARISONS!!! • Is the program effective? Effective, compared to what • Are the clients getting better? better compared to what • Are the obj/outcomes being accomplished? Accomplished, compared to what • Has ________ improved? Improved, compared to what? • Has __________ been reduced? Reduced, compared to what • These questions cannot be answered unless one compares them to something

3. Last week we focused on Non-Experimental and Pre-experimental designs. Although neither design proves that it was the program that caused the difference, pre-experimental comes closer, by ruling out more other possible causes than non-experimental. Non-experimental pre-experimental quasi-experimental experimental No time ordertime order variable time order variablestime order var Single group no manip of I.V. manipulated IV manipulate IV No control grp single group control group or control control group self-control pre/post thru repeated measures no random assignment random assignment RESEARCH done After program completion 0 X 0 LOW INTERNAL VALIDITY HIGH Answers ?#2 Did the clients in the program Get better, after they were in the program? X 0 Answersd ?#3 Did the clients get Better because They were in the Program? Answers ?#1 Did the clients in The Program Get better ? C C

4. Here are five ways to set comparison standards • I can compare the clients on a pre-established ACCEPTABLE MEASURE OF AN OBJECTIVE – a historical standard. THIS MIGHT BE CALLED AN OUTCOME STANDARD! This is often based onn best practices or what the program has done in the past. • I can compare the clients on SOCIAL INDICATORS with other parts of the area, state, country. • I can compare clients on OUTCOME VARIABLES (DEPENDENT VARIABLES), INDICATORS OR STANDARDS with a SET OF STANDARDS set by an organization, federal or state mandates, best practices, funding sources. For example. “students will have scored 400 on the MAT” • I can compare clients on OUTCOME VARIABLES (DEPENDENT VARIABLES), INDICATORS OR STANDARDS established in the review of literature. For example, for a substance abuse program: “50% of all program graduates had not relapsed at 12 month follow, based on monthly urine screens.” literature reviews ar often a good way to get or develop standards. What have other successful programs used? Is the question that the lit/ review answers. • You can get with the stakeholders and ‘set your own’ based on ‘the informed opinions of the stakeholders. I.E. “We will reduce violence in the school by 2% a year”.

5. In the non-experimental designs, there is no evidence that the clients got better WHILE they were in the program. However, using a comparison standard at least allows you to show that they have met some standard of achievement. • X 0 • C • 2. In the pre-experimental designs, usually because of cost, ethical issues or convenience considerations, there is no control group and therefore there is also no randomization. There is a demonstration of change, while in the program because of time ordering. A design of this type would look like this: • O X O • (pre and post) • C Post test only Comparison standard Comparison standard

6. The less internal validity you have, the more you might have to rely on “triangulation” • An Example: I have a non-experimental design that looks like this. • Lets say I have a reading literacy program, but have no pre-test measures. • I then compare my post-test outcomes with a literacy standard that says 90% of all children should score above___ on the___. • But I do not know if it was the program that was responsible. • I could Program outcome measure X 0 Comparison standard

7. Use focus groups with parents to explore how they think the program has helped • Follow up surveys with parents • Selected in-depth interviews with parent • Client satisfaction surveys. Any of these could be used to ‘point toward’ the program’s Effectiveness (or not) in achieving the outcome scores

8. This week we will focus on two designs that come closer to answering whether it is the program that is responsible for the change. Non-experimental pre-experimental quasi-experimental experimental No time ordertime order variable time order variablestime order var Single group no manip of I.V. manipulated IV manipulate IV No control grp single group control group or control control group self-control pre/post thru repeated measures no random assignment random assignment RESEARCH done After program completion 0 X 0 LOW INTERNAL VALIDITY HIGH Answers ?#2 Did the clients in the program Get better, after they were in the program? X 0 Answersd ?#3 Did the clients get Better because They were in the Program? Answers ?#1 Did the clients in The Program Get better ?

9. 2. In a quasi-experimental design, there is a control group, - so there is manipulation of the I.V. - but the researcher does not use randomization. For example: O X O A (group A gets program) -------------- O O B (group B does not) In quasi-experimental, Because there is manipulation of the I.V., we can be pretty certain that if there is a change treatment group and no change in the control group, the change was caused by the treatment. Notice that the quasi design makes two comparisons, pre and post; program group with non-program group. So you have answered two questions. 1. do the clients get better after they are in the program? 2. do the clients get better compared to those who are NOT in the program. Notice also that we are ‘closing in’ on showing that it was the program that caused the change by providing these two comparisons

10. This design works well with naturally occurring control groups • Comparing number of falls and increased mobility and strength in a nursing home that has started an exercise program with a nursing home that has none. • Comparing a new behavioral management program in the early grades of one school to the early grades in a school that does not have such a program. Some programs that have limited numbers of clients, use their waiting list as a control group

11. However, because there is NO RANDOM ASSIGNMENT, we cannot be ABSOLUTELY certain that the groups were equivalent. • In the previous examples, perhaps one nursing home has better exercise staff or stronger residents. Perhaps one school has children who are more able to pick up on social cues. • Although the pre-test will allow you to see how similar they are, on the outcome measures, before the program, they will not pick these other factors up. • Therefore the change could possibly be due to the INITIAL DIFFERENCES IN THE TWO GROUPS ON SOME UNKNOWN EXTRANEOUS VARIABLE.

12. Sometimes it is not technically feasible or ethically allowable to use a control group! • There is another quasi-experimental design to consider

13. Longitudinal Designs or time series designs entail the repeated study and measurement of variables over time. Generally, the same variables are measured again and again at predetermined intervals using the same instruments. In longitudinal studies, you are interested in studying change. But more than that, when did it occur, and what else was going on (associated with it) at the time... 000000000 X 000000000

14. For example, the single group pre-test post-test • 0 X 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 • By adding multiple measures, we can • Be sure that change has NOT occurred before the program • Make sure that the pre-test measures are stable • Choose to start the program –at a particular time • only after we are sure that the pre-test measures are stable • (manipulating the IV) • 4. Identify ongoing patterns and precise points where change • begins to occur. And now we have a classic time series design

15. What follows are the research designs for the 8 types of comparisons above!!!!! Symbols: R = randomization (random assignment of Ss to conditions...) O = The only measurement. Of the problem O1 = the first measurement of the problem O2. O3 = the second measurement, the third measurement and so on. X = the program or the treatment #1 LONGITUDINAL BEFORE AND AFTER O1, O2, O3, X O4, O5, O6. #2 LONGITUDINAL BEFORE, DURING AND AFTER O1, O2, O3, O4, O5, 06, O7, O8, O9 XXXXXX NOTICE THAT DESIGNS 1 AND 2 WOULD LOOK SOMETHING LIKE THIS: YOU HAVE A PROGRAM OBJECTIVE in a case management program that attempts to significantly reduce the number of psychiatric hospitalization and days hospitalized for participants. Quasi-experimental = good control for threats to internal validity important

16. Design is particularly effective if I can choose when to start the program Lets say I want to evaluate the effectiveness of a new case management Program designed to keep those with a dx of schizophrenia out of a hosptial Somewhere, in the records, you would be able to find the number of hospitalizations and days hospitalized in the previous year. It is likely that you could create two ‘monthly numbers’ ‘# of hosp’. and ‘days in hosp’. for each program participant and create a table that looks like this. Before prog during and after pro January February March April May June Client # H DiH # H DiH # H DiH # H DiH # H DiH # H DiH #h = number of hospitalizations. DiH = days in hospital In other words, you would take the data from the records, then put it in some usable form that will be easy to locate, easy to understand and easy to use/analyze. Thus I might take multiple measures on every client for several months before the Start of the program and for several months during and afterward. But I would begin the program only after I was sure of the pre-test data! Then I would compare this group of clients on # of hospitalizations and days in hospital before, during and after the program