Evaluation research

Evaluation research Using research methods in combinations Policy analysis

Evaluation • Examining the effects of programs, policies and laws • Programs: providing services to clients • Policies: rules for handling clients, actions taken i.e., policy on deadly force for fleeing felons • Laws: what are the real impacts

Evaluation • People assumed that particular effects would take place • However, those effects may or may not happen • There may be unanticipated side effects to a change, both good and bad

Policy • Usually some issue is raised • Demands are made, and there is support and opposition • Goals should be formulated and ways established to attain those goals (objectives) • Policy output: the program, resources allocated

Policy • Then the impact is determined • Did the changes in policy outputs have an impact on the problem? • Hypothesis: If some policy action is taken, then we expect a particular result • To test the hypothesis, must do an evaluation

Evaluation components • Process evaluation and impact • Process • Needs assessment and monitoring part of the process evaluation • The first step should be needs assessment • Frequently this is not done

Needs assessment • Is the program/policy/intervention needed? • Is the problem serious? Will it “get better” on its own? • Who are the members of the target population? (those most in need of services, deterrents, etc.) • What are their characteristics?

Needs assessment • Programs will have little impact if they don’t affect the target population • Programs will have little effect if they do not take into account the characteristics of the target population

Needs assessment • i.e., a school based program will not help chronically truant delinquents • Adding 11th and 12th grade classes at a reform school will not be useful if very few delinquents at the school are functioning at even a 9th grade level • Welfare to work requires child care

Needs assessment • There is a need to gather information to determine the characteristics of the target population • May not be readily apparent, as the target population might not be in the system • Assessment methods: records, key informants, gatekeepers, data collection

Needs • If the target population is not reached, even good programs, policies and laws may not have an impact • Weed and Seed example • Adolescent drug offender example

Needs • Establishment of goals and objectives • Exercise: what are the goals of a prison • Mechanisms (objectives, activities) for meeting those goals • Develop a list

In an evaluation • The research must ask • What does the program intend to accomplish? • How do existing staff determine whether they have attained their goals? • Have formal goals and objectives been identified and prioritized?

Evaluation • What performance measures are currently used? • Are they adequate, or must additional measures be developed as part of the evaluation? • It would be very useful if evaluators were included at the start of a program to establish measures, seldom happens

Monitoring (process) • What is actually happening in the intervention? • What happens on paper, theoretically, may be different from what actually happens • Who is actually served by the program? • Are they members of the target population?

Monitoring • If they are not, the program will probably have little impact • how are people selected for inclusion, or how do they decide to participate? • How do participants differ from members of the target population? • How do potential participants differ from nonparticipants?

Monitoring • How do staff feel about the program or policy? • Example: laws and prosecutors • Juvenile judges and juvenile justice programs • Is the program or policy actually being implemented?

Monitoring • Are all the components being addressed? • Example of drug courts • Program components and benchmarks, means of assessing whether program components are being met

Monitoring • Are there problems that were not anticipated? • i.e., lack of community support, resistance to the program or policy, reactions of criminal justice personnel, unexpected obstacles

Monitoring • Unanticipated side effects • Could be positive or negative • Positive: conditions improve for people not in the target population • Mental health centers, nutrition requirements for WIC • Negative: example of net widening • Program may do something else

Evaluation research