Research Methods in Crime and Justice

1. Research Methods in Crime and Justice Chapter 13 Evaluation Research Methods

2. Evaluation Research Methods • Evaluation research has become a critical part of criminal justice administration. • The analysis of a program’s efficiency and effectiveness is an important part of a criminal justice practitioner’s job. • Evaluation research considers the effectiveness, efficiency, and the unintended consequences of a program, process or policy.

3. Evaluation Research Basics • Like the qualitative and non-reactive research traditions, evaluation research is not a single methodology. • Instead, evaluation research is any applied research method that considers the effectiveness, efficiency, and the unintended consequences of a program, process or policy.

4. Types of Evaluation Research • There are two commonly used methods in evaluation research. • A cost/benefit analysis refers to a technique that compares the costs and benefits of an existing or future program, process or policy. • The multiple methods approach involves the use of multiple data gathering techniques to evaluate a program, process or policy.

5. The Benefits of Evaluation Research • Evaluation research is effective at; • Demonstrating the effectiveness or efficiency of an existing program, process or policy. • Monitoring the effectiveness or efficiency of new programs, processes or policies, • Evaluating the unintended consequences of a program, process or policy, and/or • Informing the debate on controversial programs, processes and policies.

6. The Limitations of Evaluation Research • Evaluation research; • Can evaluate but cannot determine the underlying outcome or objective of a program, process or policy. • Is pointless when a program, process or policy is mandatory or legally required.

7. The Evaluation Research Process • A Case Study in Evaluation Research (The Police Protective Custody Process) • In most states, there is a process by which the police can remove abused and/or neglected children from their homes and place them in temporary housing. • This process is expensive and in most cases results in considerable physical and psychological disruption. • Withrow (2001) and Withrow and Bolin (2003) were asked to conduct an evaluation of the Police Protective Custody process in Kansas.

8. The Evaluation Research Process • Asking a Research Question in Evaluation Research • The research question in evaluation research is whether the program or policy is performing as it was intended to. • Is it effective? • Is it efficient? • Are their unintended consequences or opportunities?

9. The Evaluation Research Process • Asking a Research Question in Evaluation Research – cont’d • The PCC evaluation had the following research objectives (questions). • To document the PPC process. • To determine what factors affect a child’s length of stay. • To make recommendations for reducing the number of PPC referrals. • To make recommendations for reducing a child’s length of stay (Withrow, 2001 p. i).

10. The Evaluation Research Process • Conducting a Literature Review in Evaluation Research • The primary literature in evaluation research is other evaluation studies on the same topic. • Many of these studies are in the possession of criminal justice and allied agencies.

11. The Evaluation Research Process • Conducting a Literature Review in Evaluation Research – cont’d • There were actually two versions of the PPC report. • The scholarly journal included literature from scholarly sources, most notably research on the effect of separation during the PPC process. • The version written for the Wichita Children’s Home included only a nominal amount of scholarly literature because it was intended to be consumed by practitioners.

12. The Evaluation Research Process • Refining the Research Question in Evaluation Research • Most evaluation research projects do not include formal hypotheses. • Instead they ask; • Did the policy, process or program have its intended effect? • Was the policy, process or program implemented efficiently? • Did the policy, process or program result in unintended consequences?

13. The Evaluation Research Process • Refining the Research Question in Evaluation Research – cont’d • Explicit hypotheses were not developed for the PPC study. • Instead Withrow and Bolin made several predictions about the relationships between the variables used in the study. • These predictions enabled then to at least begin the analysis.

14. The Evaluation Research Process • Defining Concepts and Creating Measures in Evaluation Research • In evaluation research, conceptualization and operationalization involve developing measures for policy or program success, efficiency and/or effectiveness. • Evaluation researchers must be mindful that stakeholders may have a different view on how to measure a program’s success.

15. The Evaluation Research Process • Defining Concepts and Creating Measures in Evaluation Research – cont’d • Many of the concepts in the PPC study (e.g. foster care, respite, etc.) were defined by law. • Most of the other variables (e.g. parents drug use) were dichotomous (i.e. yes or no). • This is common in criminal justice research.

16. The Evaluation Research Process • Defining Concepts and Creating Measures in Evaluation Research – cont’d • There were competing perspectives on what constitutes the success of the PPC process. • Police officers associated success with the ability to refer a child to the WCH without restriction. • Social workers defined success in terms of whether they were able to find a suitable long term living situation for the child. • Prosecutors defined success in terms of their ability to convince a judge to approve a child in need of care petition. • Representatives from social services agencies who were concerned about expenditures were defined success as the ability to limit services to children who did not need them.

17. The Evaluation Research Process • Defining Concepts and Creating Measures in Evaluation Research – cont’d • To overcome the inevitable conflict associated with alternative views of ‘success’ the researchers chose to focus on producing accurate descriptions of how the system worked. • This, they believed, would enable policy makers to make more informed decisions

18. The Evaluation Research Process • Designing a Method in Evaluation Research • The actual method or methods used by evaluation researchers depends on the type of research question asked. • The use of multiple methods has become particularly popular in evaluation research.

19. The Evaluation Research Process • Designing a Method in Evaluation Research – cont’d • Because the PPC had four objectives, no single research method would have sufficed. • The researchers used a multiple methods in our evaluation. • focus group • Secondary analysis • Archival analysis • Interviews

20. The Evaluation Research Process • Designing a Method in Evaluation Research – cont’d • Focus groups with representatives from the affected agencies allowed us to; • Document the PPC process, and • Identify redundancies in the process. • Secondary analysis of data on admitted children allowed us to identify the factors that affected; • A child’s length of stay, and • Whether a child would be admitted to state custody.

21. The Evaluation Research Process • Collecting Data in Evaluation Research • The process by which evaluation researchers collect data depends on the specific methods they use. • There are a two important considerations specific to evaluation research. • Gaining full access to data during an evaluation is sometimes difficult. • Criminal justice data is notoriously incomplete, inaccurate, and inconsistent.

22. The Evaluation Research Process • Collecting Data in Evaluation Research – cont’d • In the PPC study the focus group required considerable coordination. • The practitioners involved in the PPC process are busy. • So finding a single day when they all could meet was a rather daunting task. • The data set for the secondary analysis included information on 6,600 children. • This data set had to be ‘cleaned’ to facilitate the analysis. • In order to avoid the possibility of violating a child’s privacy, the data set did not contain (at the researcher’s request) names and other personal identifiers.

23. The Evaluation Research Process • Analyzing and Interpreting Data in Evaluation Research • The analyzes and interprets of the evaluation’s results depends on; • the nature of the research methods used, • the data produced by the research methods, and • the overall objectives of the study.

24. The Evaluation Research Process • Analyzing and Interpreting Data in Evaluation Research – cont’d • The PPC study produced (for the first time) a flow chart of the PPC process that enabled policy makers to ‘see’ system redundancies. • The secondary analysis found that children were more likely to be taken from their parents if; • they had previously received state social or welfare services, • they had parents who used illegal drugs, • they had previously been abused, and/or • they had limited parental supervision.

25. The Evaluation Research Process • Analyzing and Interpreting Data in Evaluation Research – cont’d • The PPC study made two primary recommendations. • The PPC process should be managed by a team of professionals. • Police officers should only use the PPC process for children who; • are being or have been abused or neglected, • have received any type of social or welfare services, • Have limited parental supervision, and • Are younger than 16 years of age.

26. The Evaluation Research Process • Communicating the Findings from Evaluation Research • Most evaluation studies are published ‘in-house’. • Prior to the release of the report it is prudent to allow affected individuals to review the report for accuracy. • Often evaluation researchers are asked to present their findings in multiple formats to different audiences. • On occasion the results of an evaluation report will cause a public controversy.

27. The Evaluation Research Process • Communicating the Findings from Evaluation Research - cont’d • The initial report (for agency consumption) was brief to accommodate busy practitioners. • Another version of the report was published in a scholarly journal. • In both situations the reports clearly identified the limitations of the study.

28. Getting to the Point • Evaluation research is designed to determine whether a program, process or policy is achieving its intended outcome and/or resulting in any unintended consequences. • Evaluation research is becoming increasingly critical to criminal justice practice.

29. Getting to the Point • Evaluation research typically involves either; • a cost/benefit analysis, in which the costs associated with a policy or program are weighed against the benefits, or • a multiple methods approach, in which various data collection methods are used to evaluate a policy or program.

30. Getting to the Point • Evaluation research is effective at demonstrating and monitoring the effectiveness or efficiency of new and existing programs, processes or policies. • It can also identify the unintended consequences of and provide objective information about controversial programs, processes and policies.

31. Getting to the Point • Evaluation research is not effective; • without agreed upon objectives, or • when a program is mandated by law, policy or higher authority.

32. Getting to the Point • Evaluation research can be exploratory, descriptive or explanatory depending on the researcher’s objective. • It is almost always applied because the researcher usually intends for the results to influence policies and programs.

33. Getting to the Point • Although evaluation research reports can be found in scholarly journals, the overwhelming majority of them are in the possession of; • various government agencies, • privately sponsored research organizations, and • individual criminal justice agencies. • Such reports are typically available through the internet or can be made available through a direct request to the individual agency in possession of the report.

34. Getting to the Point • Evaluation research projects may begin with general research questions or explicit hypotheses depending on the nature of the project. • In either case, the primary purpose is to investigate whether a program, process or policy is having its intended results.

35. Getting to the Point • In evaluation research, researchers typically start by defining and operationalizing policy or program effectiveness. • Because criminal justice agencies are publically funded, the researchers who evaluate criminal justice programs should be mindful of the various political objectives when developing measures of program effectiveness.

36. Getting to the Point • The actual method or methods used by evaluation researchers to collect data depends on the type of research question(s) asked. • The use of multiple methods has become particularly popular in evaluation research.

37. Getting to the Point • Gaining access to data during evaluation research is sometimes difficult. • In many cases, individuals who make the policy or work in the program are less than willing to give information, lest the policy or program be found to be unsuccessful or inefficient.

38. Getting to the Point • Criminal justice data is also notoriously incomplete, inaccurate and inconsistent. • As a result, evaluation researchers should expect that they will need to spend considerable time preparing the data for analysis.

39. Getting to the Point • The interpretation process in evaluation research is not much different than that required in any research project. • But evaluation researchers should bear in mind that criminal justice data are notoriously incomplete and inconsistent. • This limitation should always been acknowledged in the interpretation of the results.

40. Getting to the Point • Evaluation researchers are often asked to present their findings in multiple formats and to multiple audiences. • When the media get involved with disseminating the results of the research, evaluation researchers should be mindful of the complicated dynamics of public opinion and messaging.

41. Research Methods in Crime and Justice Chapter 13 Evaluation Research Methods