1 / 47

Program Evaluation Designs

Program Evaluation Designs. Programs for Class Discussion. Counseling program for prison inmates Stress reduction training for employees Study skills for at-risk college students Parental training for problem children. Why Should I Care About Research?.

ardice
Download Presentation

Program Evaluation Designs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Program Evaluation Designs

  2. Programs for Class Discussion • Counseling program for prison inmates • Stress reduction training for employees • Study skills for at-risk college students • Parental training for problem children

  3. Why Should I Care About Research? • Answering questions and making decisions • We encounter research every day • Common sense is often wrong

  4. While You’re on Vacation, Burglars Go to Work According to FBI statistics, over 26% of home burglaries take place between Memorial Day and Labor Day. To help you protect your home, we are offering a special summer sale price for installation of a home security system. Home Safety Experts Route 11 and Wayland Road 555-6676

  5. How Do I Know What to Research? • Ideas • Hypotheses - well thought-out suggestions or ideas • Theories - systematic sets of assumptions regarding the nature and cause of particular events

  6. Social Loafing Example

  7. Aggression Example

  8. How Do I Find Previous Research? • Written sources • Journals • Trade Magazines • Magazines • Books • Electronic Resources • First Search • Psych Info • InfoTrac • The Web

  9. Where Will I Conduct Research? • Locations • Laboratory • Field • Office • Issues • External validity (generalizability) • Control

  10. What Research Method Should I Use?Independent and Dependent Variables • Independent Variable • Experimental group • Control group • Dependent Variable

  11. A researcher thinks that smaller groups will be more cohesive than larger groups Independent variable = Group size Dependent variable = Level of cohesion Number of Group Members

  12. A researcher thinks that asking for a large request and then a smaller request will increase the chance of a person agreeing to the smaller request Independent variable = Size of initial request Dependent variable = % agreeing to small request Experimental Condition

  13. Identify the Independent and Dependent Variables • Are people from large families more likely to commit crimes than people from small families? (A) • Will taking an SAT prep class increase scores on the SAT? (B) • Will taking Vitamin A decrease night blindness? (c)

  14. Identify the Independent and Dependent Variables • A researcher found that a higher percentage of serial killers have head injuries than do nonviolent criminals (d) • A researcher found that people who set goals perform better at work than people who do not set goals (e) • A researcher found that the amount of time that passes between viewing a crime and reporting a crime influences the number of errors made by eyewitnesses (f)

  15. Evaluation of Program ResultsCriteria • Content validity • Participant reactions • Participant learning • Behavioral change • Bottom-line measures

  16. Dependent Measures in Program Evaluation • Did people know about the program? • Did people participate? • Did they stay in the program?

  17. Dependent Measures in Program Evaluation • Did they change? • Reality versus perception • Measures • Attitudes • Behavior • Skills • Health • How much of a change? • Any change • Enough to meet some criterion

  18. Dependent Measures in Program Evaluation • What did they think about the program • Process • Effectiveness

  19. What Research Method Should I Use?Research Methods • Experiment • independent variable is manipulated {and} • Subjects are randomly assigned to conditions • dependent variable • Quasi-experiment • Independent variable is not manipulated {or} • Subjects are not randomly assigned to conditions • Case study • Survey • Interview • Natural observation

  20. Evaluation of Program ResultsSingle-Group, Nonexperimental Designs Program Posttest

  21. Evaluation of Program ResultsSingle-Group, Nonexperimental Designs Pretest Program Posttest

  22. Threats to Internal Validity • Changes Unrelated to the Program • Maturation • History • Participant Issues • Selection • Attrition • Regression toward the mean • Data Collection Issues • Testing • Instrumentation

  23. Construct Validity in Pretest-Posttest Designs • Of concern when self-report measures are used • Types of self-reported change • Alpha (real change) • Beta (respondents change their understanding of the scale) • Gamma (respondents change their understanding of the variable being measured) • Examples • Asking students about school safety • Asking couples about their marital satisfaction

  24. ExampleNew Child Care Center • 2000 Employee absenteeism rate = 5.09% • 2001 On-site child-care center established (Jan 1) • 2001 Employee absenteeism rate = 3.13%

  25. Demonstrating Effects in Designs Without a Control Group • Means • Show means for the periods before and after intervention • Use a t-test to determine statistical significance • Report d to show practical significance (Mafter – Mbefore) ÷ SDoverall • Time Series Analysis • Looks at trends over time • Interrupted time series involves an intervention • Most powerful is ABAB design • Most common is AB design

  26. Child Care Center Example Child Care Center Started

  27. Child Care Center Example Child Care Center Started

  28. Child Care Center Example Child Care Center Started

  29. Conformity & SuicidePhillips (1980)

  30. Conformity and Commercial Airline Fatalities(Phillips, 1983)

  31. Time Series Charts • Include actual plotted data • Include Lowess lines • Smoothed data • Plotted point is the median of the 3 adjacent points • Should include notations for other important events

  32. Control Group Designs • Control for many of the threats to internal validity • Groups need to be as similar as possible • Using multiple groups increases accuracy of interpretation • Placebos are often used for the control group • Control group treatment • Never get treatment • Get treatment later

  33. Evaluation of Program ResultsControl Group Designs Pretest Program Posttest Pretest Posttest

  34. Evaluation of Program ResultsSolomon four-groups design Program Posttest Group 1 Pretest Program Posttest Group 2 Pretest Posttest Group 3 Posttest Group 4

  35. Assigning Subjects to Conditions • Two Choices • Independent groups • Repeated measures • Considerations • Number of available subjects • Order effects • Will participating in one condition affect responses in another

  36. Where Do I Get My Subjects?Who Will Participate? • Size • Students vs. “real world” Does it Matter? If you were investigating whether the type of evidence (e.g., eyewitness testimony, DNA, fingerprints) influenced jurors decisions, would students as subjects be different from having people from the community?

  37. Where Do I Get My Subjects?Sampling • Types of Samples • Random • Representative • Non-random/representative • Sampling Methods • Random selection • Simple • Block randomization • Proportionate • Convenience • Random assignment

  38. Randomized Block Designs • Used when: • there are several conditions • you want equal numbers in each condition • You are concerned about time-related variables • Example • 4 conditions (A, B, C, D), 8 subjects per block • Number of blocks = the number of subjects desired for each condition • Random blocks might be CBAD, DABC, BCDA, ACDB, CEBA, DBCA, BCAD, ADBC,

  39. What Type of Sampling Method is Being Used? • A researcher has the students in her classes fill out a questionnaire • A researcher gives $6 to people who will participate in his study. As the people arrive, he flips a coin to see if they will be in the experimental or the control condition. • A manager wants to see if a training program will increase performance. She selects every third name from the company roster to participate. Employees with an odd number at the end of their social security number are given one training program and those with an even number are given another.

  40. Where Do I Get My Subjects?Inducements to Participate • Extra credit • Money • Intrinsic reasons • Ordered to participate Does it Matter? Would the inducement used affect the type of person agreeing to participate? In what ways?

  41. Where Do I Get My Subjects?Informed Consent • Ethically required • Can be waived when • Research involves minimal risk • Waiver will not adversely affect rights of participants • Research could not be done without the waiver

  42. Is Informed Consent Needed? • An experimenter wants to study the effects of electric shock on reducing patients’ depression levels • A researcher wants to conduct a telephone survey in which she asks people their five favorite TV shows. She will then determine if males and females like different shows. • A researcher wants to determine the types of people who litter. He plans to hide above a road and record information about the people who litter or don’t litter (e.g., age, sex, type of car).

  43. Running the Study • Informed consent • Instructions • Task completion • Deception? • Debriefing

  44. External Validity • Can our findings be generalized to other situations? • Considerations • Sample • How realistic was the task? • How realistic was the dependent variable? • Has the study been replicated in other situations?

  45. Putting it all Together Redesign Your Study

More Related