Designing Designs for Research

1. Designing Designs for Research

2. Designing Designs • How do you establish cause? • How do you minimize threats to validity? • How do you construct a design? • What is a good design?

3. How Do You Establish Cause? Temporal Precedence

4. How Do You Establish Cause? Temporal precedence then Cause Effect Time

5. How Do You Establish Cause? Temporal precedence then Cause Effect Time Why is this important?

6. How Do You Establish Cause? Temporal precedence then Cause Effect Time Why is this important? Cyclical functions Inflation Unemployment

7. Cause Effect then Time How Do You Establish Cause? Temporal Precedence Covariation of Cause and Effect

8. Cause Effect then time How Do You Establish Cause? Temporal precedence Covariation of cause and effect If X, then Y. If not X, then not Y.

9. Cause Effect then time How Do You Establish Cause? Temporal precedence Covariation of cause and effect If X, then Y. If not X, then not Y. If program given, then outcome observed. If program not given, then outcome not observed.

10. Cause Effect then time How Do You Establish Cause? Temporal precedence Covariation of cause and effect If X, then Y. If not X, then not Y. If program given, then outcome observed. If program not given, then outcome not observed. If more of program, then more of outcome observed. If less of program given, then less of outcome observed.

11. Cause Effect then time How Do You Establish Cause? Temporal precedence If X, then Y. If not X, then not Y. Covariation of cause and effect No alternative explanations

12. Cause Effect then time How Do You Establish Cause? Temporal precedence If X, then Y. If not X, then not Y. Covariation of cause and effect No alternative explanations causes? Program Outcome

13. Cause Effect then time How Do You Establish Cause? Temporal precedence If X, then Y. If not X, then not Y. Covariation of cause and effect Alternative cause No alternative explanations Alternative cause Causes? Program Outcome Alternative cause Alternative cause

14. In Typical Outcome Evaluation • Is taken care of because you intervene before you measure outcome • Is taken care of because you control the intervention • Is the central issue of internal validity -- usually taken care of through your design Temporal Precedence Covariation of Cause and Effect No Alternative Explanations

15. How Do You Minimize Threats to Validity? • By argument • By measurement or observation • By design • By analysis • By preventive action

16. How Do You Minimize Threats to Validity? By argument

17. How Do You Minimize Threats to Validity? By argument Simply say the threat isn’t reasonable.

18. How Do You Minimize Threats to Validity? By measurement

19. How Do You Minimize Threats to Validity? By measurement Measure that the threat didn’t happen.

20. How Do You Minimize Threats to Validity? By design R O X O Maturation threat?

21. How Do You Minimize Threats to Validity? By design R O X O R O O To address a maturation threat Add a group to control for the threat.

22. How Do You Minimize Threats to Validity? By analysis Mortality threat : Are your dropouts different from those retained in your study?

23. How Do You Minimize Threats to Validity? By analysis Pretest means Show through analysis that the threat isn’t plausible.

24. How Do You Minimize Threats to Validity? By preventive action Rivalry?

25. How Do You Minimize Threats to Validity? By preventive action Get program next semester Rivalry Don’t allow the threat to happen in the first place.

26. How Do You Construct a Design? • Across time • Across programs • Across observations • Across groups Expand...

27. How Do You Construct a Design? Start with the simplest design.

28. How Do You Construct a Design? Start with the simplest design. X O

29. How Do You Construct a Design? Expand across time.

30. How Do You Construct a Design? Expand across time. O O X O O X O O O

31. How Do You Construct a Design? Expand across programs.

32. How Do You Construct a Design? Expand across programs. O X1 O O X2 O O X3 O Program variations

33. How Do You Construct a Design? Expand across observations.

34. How Do You Construct a Design? Expand across observations. O1 X O1,2,3 O1 O1,2,3 Measurement variations

35. How Do You Construct a Design? Expand across groups. R O X O R O O

36. How Do You Construct a Design? Expand across groups. R O X O R O O R O X O R O O Add a replicate.

37. How Do You Construct a Design? Expand across groups and time. R O X O R O O N O

38. How Do You Construct a Design? Expand across groups and time. R O X O R O O N O

39. How Do You Construct a Design? Expand across groups and time. R O X O R O O N O Groups Time

40. How Do You Construct a Design? X O Start

41. How Do You Construct a Design? X O Start R O X O O R O O X O N O X O N O O Expand

42. How Do You Construct a Design? X O Start R O X O O R O O X O N O X O N O O Expand R O X O R O O N O O Contract

43. What Is a Good Design? • Theory-grounded • Situational (adapted) • Feasible • Redundant • Efficient