Practice- Based Evaluation Research Week 4 Day 2

1. DIE 4564 Research Methods Practice- Based Evaluation Research Week 4 Day 2

2. Chapter Outline • Topics Appropriate for Evaluation Research • Formulating the Problem: Issues of Measurement • Types of Evaluation Research Designs • The Social Context • Social Indicators Research • Ethics and Evaluation Research • Quick Quiz

3. Evaluation Research • Evaluation Research – research undertaken for the purpose of determining the impact of some intervention, such as a program aimed at solving a problem.

4. Topics Appropriate for Evaluation Research • Needs assessment studies – studies that aim to determine the existence and extent of problems, typically among a segment of the population.

5. Topics Appropriate for Evaluation Research • Cost-Benefit Studies – studies that determine whether the results of a program can be justified by its expense (both financial and other). http://www.youtube.com/watch?v=Qt8HNNWIV-c

6. Topics Appropriate for Evaluation Research • Monitoring Studies – studies that provide a steady flow of information about something of interest, such as medical error rates or the outbreak of an epidemic. • Six Sigma – management philosophy that uses data to eliminating mistakes, waste, and re-work. • Sigma = standard deviation • Six = goal level of perfection • Based on measuring, analyzing, improving, and controlling

7. Topics Appropriate for Evaluation Research • Program Evaluation Studies – the determination of whether a social intervention is producing the intended result. • Also called Outcome Assessment Studies

8. Topics Appropriate for Evaluation Research • Review Question • A researcher wants to determine if changing what is stocked in school vending machines will improve nutrition intake. This is an example of ______ research. A. needs-assessment B. cost-benefit C. outcomes assessment D. survey

9. Topics Appropriate for Evaluation Research • Review Question • A researcher wants to determine if changing is what stocked in school vending machines will improve nutrition intake. This is an example of ______ research. • The correct answer is C. outcomes assessment

10. Formulating the Problem: Issues of Measurement • Specifying Outcomes • Measuring Experimental Contexts • Specifying Interventions • Specifying the Population • New versus Existing Measures • Operationalization Success/Failure

11. Formulating the Problem: Issues of Measurement • Review Question • Which one of the below is not one of the measurement issues in evaluation research? A. Defining the units of analysis B. Measuring the outcome C. Measuring the stimulus or intervention D. Defining success E. Defining failure

12. Formulating the Problem: Issues of Measurement • Review Question • Which one of the below is not one of the measurement issues in evaluation research? • The correct answer is A. Defining the units of analysis

13. Types of Evaluation Research Design • Experimental Designs

14. Types of Evaluation Research Design • Quasi-Experimental Designs – nonrigorous inquiries somewhat resembling controlled experiments but lacking key elements such as pre- and post-testing and/or control groups.

15. Types of Evaluation Research Design • Quasi-Experimental Designs • Time-Series Design – a research design that involves measurement made over some period. • Nonequivalent Control Groups – a control group that is similar to the experimental group but is not created by the random assignment of subjects. • Multiple Time-Series Designs – the use of more than one set of data that were collected over time, so that comparisons can be made.

16. Types of Evaluation Research Design Time-Series Design

17. Types of Evaluation Research Design Time-Series Design

18. Types of Evaluation Research Design Time-Series Design

19. Types of Evaluation Research Design • Review Question • True or False: Evaluation research requires quantitative analysis of collected information.

20. Types of Evaluation Research Design • Review Question • False: Evaluation research can use both qualitative and quantitative analyses of collected information.

21. The Social Context • Logistical Problems – getting subjects to do what they are supposed to do. • Evaluation research occurs within the context of real life. • Discuss example: Motivating people with diabetes • Discuss example: Administrative control of break-room snacks

22. The Social Context • Use of Research Results • Why evaluation research results are not always put into practice: • Implications may not be presented in a way that is understandable to the non-researcher. • Results may contradict deeply held beliefs. • Researchers may have a vested interest in the results.

23. The Social Context • Review Question • ________ refers to getting research subjects to do what they’re supposed to do.

24. The Social Context • Review Question • Logistics refers to getting research subjects to do what they’re supposed to do.

25. Social Indicators Research • Social indicators – measurements that reflect the quality or nature of social life. Social indicators are often monitored to determine the nature of social change in a society.

26. Social Indicators Research • Review Question • Which of the below would not be a type of social indicators research? A. Comparing child obesity rates across states with and without schools physical fitness programs B. Comparing infant mortality rates across countries with and without nationalized health care C. Comparing injury rates before and after passage of a employee safety education law D. Comparing a student’s test grades across different classes in a semester

27. Social Indicators Research • Review Question • Which of the below would not be a type of social indicators research? • The correct answer is D.

28. Ethics and Evaluation Research • Sometimes the social interventions being assessed in evaluation research themselves raise ethical issues. • Evaluation research may entail added pressure to produce specific results, as desired by interested parties. • Fraudulent research results in an evaluation study can have severer consequences than consequences produced by other types of research.

29. Quick Quiz

30. Chapter 12 Quiz Why has evaluation research grown in popularity? A. federal requirements demanding program evaluations B. the availability of research funds to fulfill federal requirements C. both of the above D. none of the above

31. Chapter 12 Quiz Answer: C. Evaluation research has grown in popularity because of federal requirements demanding program evaluations, and the availability of research funds to fulfill federal requirements.

32. Chapter 12 Quiz _____ aim to determine the existence and extent of problems. A. Longitudinal studies B. Needs assessment studies C. Cost-benefit studies D. Monitoring studies

33. Chapter 12 Quiz Answer: B. Needs assessment studies aim to determine the existence and extent of problems.

34. Chapter 12 Quiz _____ provide a steady flow of information about something of interest. A. Longitudinal studies B. Needs assessment studies C. Cost-benefit studies D. Monitoring studies

35. Chapter 12 Quiz Answer: D. Monitoring studies provide a steady flow of information about something of interest.

36. Chapter 12 Quiz Besides making measurements relevant to the outcomes of a program, research must measure the _____. A. primary cause B. program intervention C. tertiary effect D. all of the above E. none of the above

37. Chapter 12 Quiz Answer: B. Besides making measurements relevant to the outcomes of a program, research must measure the program intervention.

38. Chapter 12 Quiz A research design that involves measurements made over some period may be defined as A. time series designs. B. quasi experiments. C. multiple time series designs.

39. Chapter 12 Quiz Answer: A. A research design that involves measurements made over some period may be defined as time series designs.

40. Chapter 12 Quiz _____ is a form of applied research. A. Evaluation B. Experimentation C. Interviewing D. Field research

41. Chapter 12 Quiz Answer: A. Evaluation is a form of applied research.

42. Chapter 12 Quiz _____ determine whether the results of a program can be justified by its expense. A. Longitudinal studies B. Needs assessment studies C. Cost-benefit studies D. Monitoring studies

43. Chapter 12 Quiz Answer: C. Cost-benefit studies determine whether the results of a program can be justified by its expense.

44. Chapter 12 Quiz A key variable for evaluation researchers is to measure the _____. A. independent variable B. dependent variable C. response variable D. none of the above

45. Chapter 12 Quiz Answer: C. A key variable for evaluation researchers is to measure the response variable.

46. Chapter 12 Quiz _____ are distinguished from “true” experiments primarily by the lack of random assignment of subjects to an experimental and a control group. A. Time series designs B. Quasi experiments C. Multiple time series designs

47. Chapter 12 Quiz Answer: B. Quasi experiments are distinguished from “true” experiments primarily by the lack of a random assignment of subjects to an experimental and a control group.

48. Chapter 12 Quiz Professor Yee wants to do an evaluation study of the effects of a patient education program on patient anxiety. He uses one wing in a hospital for the experiment and compares the results with a similar group of patients in a similar wing in another hospital. Which design would be best? A. classical B. nonequivalent control group C. time-series D. posttest-only control group design

49. Chapter 12 Quiz Answer: B. Professor Yee wants to do an evaluation study of the effects of a patient education program on patient anxiety. He uses one wing in a hospital for the experiment and compares the results with a similar group of patients in a similar wing in another hospital. The nonequivalent control group design would be best.

50. Chapter 12 Quiz Evaluation researchers encounter more logistical problems than other researchers because evaluation research A. occurs in the context of real life. B. takes longer. C. is more costly. D. has more measurement problems. E. examines more variables.