Week 7 Outcome evaluation Part 2

1. Week 7 Outcome evaluation Part 2

2. Back to these 3 questions! Each different research design allow you to answer one of these questions • did the clients in the program get better? (minimal question –usually unacceptable. Says nothing about clients in relation to the program) 2. Did the clients in the program get better, after they were in the program. (basic acceptable standard –shows people in the program improved or met P.O.s- if answered, does not show that the program was responsible for the difference. • Did the clients in the program get better because of the program (highest standard – shows that the Program was the cause of client improvement) There are 4 different types of outcome research designs

3. In outcome evaluations research designs, can be one of the these four! Moving from left to right, Each has progressively more control over internal validity, until you get to the far right and the true experiment! Non-experimental pre-experimental quasi-experimental experimental No time ordertime order variable time order variablestime order var Single group no manip of I.V. manipulated IV manipulate IV No control grp single group control group or control control group self-control pre/post thru repeated measures no random assignment random assignment RESEARCH done After program completion LOW INTERNAL VALIDITY HIGH Answersd ?#3 Did the clients get Better because They were in the Program? Answers ?#2 Did the clients in the program Get better, after they were in the program? Answers ?#1 Did the clients in The Program Get better ?

4. Both pre-experimental and quasi-experimental, occur in real time; that is the independent variable (the program ), occurs before the dependent variable (the change) in REAL TIME. In other words, it must be clear that clinets get better AFTER they aare in the program. This is called “time ordering”. The other major difference is that a. in quasi-experimental, the I.V. is Manipulates and b. in quasi-experimental there is often a control group. Non-experimental pre-experimental quasi-experimental experimental No time ordertime order variable time order variablestime order var Single group no manip of I.V. manipulated IV manipulate IV No control grp single group control group or control control group self-control pre/post thru repeated measures no random assignment random assignment RESEARCH done After program completion LOW INTERNAL VALIDITY HIGH Answersd ?#3 Did the clients get Better because They were in the Program? Answers ?#2 Did the clients in the program Get better, after they were in the program? Answers ?#1 Did the clients in The Program Get better ?

5. here are symbols for research designs. On the slide that follows we will show what each of the four designs look like and how comparisons are made O = The only measurement. Of the problem O1 = the first measurement of the problem O2. O3 = the second measurement, the third measurement and so on. X = the program or the treatment Xe = experimental treatment Xc = traditional (control treatment R = randomization (random assignment of Ss to conditions...) Also there will be a brief statement re; what comparisons are made.

6. 0 = The only measurement. Of the problem 01 = the first measurement of the problem 02. O3 = the second measurement, the third measurement and so on. X = the program or the treatment, Xe = experimental treatment, Xc = control tx . R = randomization (random assignment of Ss to conditions...) Non-experimental pre-experimental quasi-experimental experimental No time ordertime order variable time order variablestime order var Single group no manip of I.V. manipulated IV manipulate IV No control grp single group control group or control control group self-control pre/post thru repeated measures no random assignment random assignment RESEARCH done After program completion LOW INTERNAL VALIDITY HIGH X 0 0 X 0 0 X 0 R 0 X 0 0 0 R 0 0 or or 01020304 X 05060708 R0Xe 0 R 0 Xc 0 comparison made comparison made comparison made to control comparison made to To standard after to self (pre/post) & (no random) or self (pre/post) control program Completion standard after with repeated measures

7. A note on non-experimental designs Non-experimental pre-experimental quasi-experimental experimental No time ordertime order variable time order variablestime order var Single group no manip of I.V. manipulated IV manipulate IV No control grp single group control group or control control group self-control pre/post thru repeated measures no random assignment random assignment RESEARCH done After program completion LOW INTERNAL VALIDITY HIGH X 0 Answersd ?#3 Did the clients get Better because They were in the Program? Answers ?#2 Did the clients in the program Get better, after they were in the program? Answers ?#1 Did the clients in The Program Get better ? Often used as a last resort

8. Sometimes when a program has already begun – or is in the process of finishing up (having clients graduate or finish), they decide to measure outcomes. • The problem is that they did not measure clients before they started the program • so now, the only thing they can find out is how clients are doing now – after they have completed the program! • This design can only answer one question!! Did the clients get better OR measure up to the outcome standard (comparison standard) X 0 C Outcome measures program Comparison standard comparison This design is more common than I would like to admit. Its biggest problem is that there is no way the program can really take credit for any change!!!

9. As bad as this design is X 0 C It is even worse without a comparison standard X 0 Here , I do not even know if the programs outcome scores have any merit at all, because there is no standard of measurement/comparison

10. You actually are experiencing something similar • We are beginning to use a “modified version” of the licensing exam for one of our program outcome measures. • In the future, students will get a ‘pre-test’ when they come into the program and a ‘post-test’ when they graduate. 0 X 0

11. However, we are starting this with your cohort • So for now, here is your design on this one outcome measure X 0 However, there is a comparison standard C That we will use in evaluating the cohort scores

12. A note on Pre-experimental designs Non-experimental pre-experimental quasi-experimental experimental No time ordertime order variable time order variablestime order var Single group no manip of I.V. manipulated IV manipulate IV No control grp single group control group or control control group self-control pre/post thru repeated measures no random assignment random assignment RESEARCH done After program completion 0 X 0 LOW INTERNAL VALIDITY HIGH Answers ?#2 Did the clients in the program Get better, after they were in the program? X 0 Answersd ?#3 Did the clients get Better because They were in the Program? Answers ?#1 Did the clients in The Program Get better ?

13. In my opinion, the easiest comparison is to compare clients with themselves, either before and after the program or before, during and after.. the important thing is that when you are looking for “change” or “difference”,. The important thing is YOU MUST HAVE A COMPARISON!!!!! 0 X 0 Post –test is a measure of outcome Achievement after completion of program Creates a Baseline Comparison A pre-test C Allows comparison of outcome achievement With a standard