CRJS 4466 PROGRAM & POLICY EVALUATION LECTURE #4

CRJS 4466 PROGRAM & POLICY EVALUATION LECTURE #4 1. Evaluation projects handout 2. Questions?

3. Evaluation Concepts • formative (process) evaluation • summative (outcome) evaluation • participatory evaluation • empowerment evaluation • evaluation questions (evaluation framework) • assessment of program theory (theory of action) • efficiency analysis

performance criterion • program theory • impact theory • service utilization plan (target access) • organizational plan • implementation failure (e.g • theory failure (e.g. ‘new Coke’; Beta vs VHS)

4. Evaluation frameworks: • Needs assessment - how? (e.g. leadership competencies) • Assessment of program theory - how? (e.g. ‘what works’) • Assessment of program process - how? (e.g. activities) • Outcome or impact assessment - how? (e.g. UFT) • Efficiency assessment - how? (e.g. Dell Computer)

5. Components of program theory: • organizational plan + service utilization plan = process theory • process theory + impact theory = program theory

6. Needs Assessment • definitions of need: • 1. normative (expert opinion based on comparison) • 2. felt need (client perceptions) • 3. expressed demand (e.g. waiting list clients) • 4. comparative need (based on service utilization in • one population, inferred to another) • 5. mandated requirements (e.g. legislation) • 6. humanitarian (moral/ethical requirement)

7. Needs Assessment • “a decision-aiding tool used for resource allocation, program • planning, and program development based on the • assumption that planned programming can alleviate • distress and aid growth” • requirement to conduct needs assessments BEFORE • programs are designed, implemented – it is the first • type or stage of program evaluation • not just needs for service – but criteria related to • service usage, actual demand, etc.

8. Planning a Needs Assessment • purpose of assessment • scope of assessment (population) • stakeholders to include • budget/resources • time frame • identify specific information required • determine availability of information/need to collect • design methodology and data collection instruments • gather and analyze data • prepare draft report • disseminate draft report to stakeholders for comment • formally disseminate results

9. Needs Assessment Approaches • secondary data analysis – including census, • ‘patterns of use/client utilization data’, other social • indicators data, provincial/municipal databases, etc. • (note validity, reliability, ecological fallacy) • ‘impressionistic approaches’ – key informants, • community forums, ‘town halls’ • nominal groups, Delphi technique and focus groups • surveys • convergent analysis/multi-method approaches • being creative • mandated requirements • public opinion

10. Community Readiness • no awareness • denial • vague awareness • preplanning • preparation • initiation • stabilization • confirmation/expansion • professionalization

11. Data Sources - Existing Statistics • public opinion polls (e.g. Environics, Gallup) • census data (e.g. poverty) • vital statistics (e.g. suicide; infant mortality; cancer) • social indicators surveys (e.g. unemployment; divorces) • social surveys (e.g. the GSS) • organization records (e.g. school records; sales figures) • statistical reports (e.g. stock prices; crime statistics)

12. Data Sources - New Information • questionnaires (e.g. competency inventories) • structured interviews (e.g. drug investigators) • focus groups (e.g. police workload) • observation (e.g. Wapekeka, NBPS) • simulation (e.g. remand counts)

13. Describing service needs • “fleshing out” the nature of needs - where and how do the • needs arise? • the importance of social context: ethnic/cultural factors, • age factors, sensitivity, social class factors (e.g. mandatory • charging practices; investigative interviewing) • need, demand, access to services

A need is “a social construction negotiated between a set of social agents with responsibility for social programs and policy and a set of claimants and their advocates who assert that a problem exist that warrants intervention” (Rossi, Freeman and Lipsey, 1999) • Within the context of program evaluation, the “primary focus of needs assessment is not on human needs broadly defined but, rather, on social conditions deemed unsatisfactory through some process of social judgement and presumed remedial by social programs” (Rossi, Freeman and Lipsey, 1999)

Politicians, advocates and moral leaders provide the impetus for policy and program change – “shoot from the hip” • Evaluators contribute in a mundane way by providing diagnostic advise • The policy and program arena is replete with ill-effects examples of “shooting from the hip” and the absence of appropriate diagnostics • 1960’s urban planners supported demolition of dilapidated buildings and relocation of poor to replacement housing • A Midwestern American suburb extrapolated federal adolescent crime statistics to their own community • Gun control has focused on legislative remedies forbidding the sale of guns to persons convicted of felony offences (Rossi, Freeman and Lipsey, 1999)

1. Assessment of need for a specific program or program proposal • 2. As a planning and decision-making tool for policymakers to determine how resources should be allocated across various service areas

Defining social problems • As indicated earlier, social problems are social constructions; social problems are not objective phenomena (Rossi, Freeman and Lipsey, 1999) • Program docs and funding proposals generally expose the social problem of the program – minimally these sources can lead to a more probing assessment of problem definition and what other perspectives might be applicable

Specifying the extent of the problem • Much more difficult to develop good estimates of problem extent than to identify and define a problem • Many social problems are largely invisible obscuring good estimation (e.g. population of abused children and characteristics of their perpetrators) • Multi-methods/ sources are necessary to estimate otherwise invisible pockets • Overestimation generally results from single-sourcing (e.g. estimation of rate of spousal abuse during pregnancy based on women in shelters) (Rossi, Freeman and Lipsey, 1999)

Using existing data sources for estimation • Some social issues can be estimated with confidence using routinely collected census data • When specific census data is not available, indirect estimates can sometimes be made if empirical relationships between available information and problem indicators are known • Undercounts are not uncommon on several variables with census data (e.g. counts of numerous ethnic groups, homeless) • Sometimes undercounts are politically unattractive to correct technically (resource implications; revision of electoral boundaries – U.S.) • In absence of good census data, and where validity of data is not well recognized, consider substantially agreed upon findings from several surveys (Rossi, Freeman and Lipsey, 1999)

Using social indicators to identify trends • On some topics data is collected periodically which can be used for trend analysis (e.g. population census information on household incomes, family size and sex) • This data, also known as social indicators, can be used to: • Estimate the size and distribution of a social problem • Alert decision-makers to whether specific social conditions are improving, worsening or stable • In a crude way, estimate the effects of social programs

“unfortunately, the social indicators currently available are limited in their coverage of social problems, focusing mainly on issues of poverty and employment, criminal victimization, national program participation, and household composition” (Rossi, Freeman and Lipsey, 1999)

Estimating problem parameters through social research • In many instances, no existing data source will provide estimates of the extent and distribution of a problem of interest • Or, good information may be available for a national or regional sample that can not be dissaggregated to a local level • The evaluator must then collect new data whilst being mindful of funds available and precision required (Rossi, Freeman and Lipsey, 1999)

Sources of problem parameter information • Agency records – the search may end here if the records are fastidiously kept although such is not usually the case; there also may be some danger in extrapolating to broader populations from a micro-population within an agency • Surveys and censuses-where no credible data exists original research may be required in the way of surveys or even complete enumerations which can both be quite resource intensive; costs can be reduced through telephone • Key informant surveys –surveying of persons whose position or experience should give them some perspective on the magnitude and distribution of the problem; validity is usually poor though; this approach is better used for info collection about population characteristics and needs but not of numbers of persons affected (Rossi, Freeman and Lipsey, 1999)

Forecasting need • Forecasting future trends can be quite risky, which is not to say that they should not be used, but that they should be reviewed critically • Forecasting techniques can be qualitative, causal modeling or time-series

Forecasting techniques • Qualitative techniques – based on judgement rather than records of past data; better track record of prediction when they are conducted thoughtfully and in a systematic manner (i.e., nominal groups, delphi groups, relevance trees) • Causal modeling – variable to be forecast is related statistically to one or more other variables which are thought to cause changes in it; nearly always this means that regressions analysis in conjunction with past data is used to establish the relationship (e.g. policies such as restricting the money supply are fed into the model to give forecasts of inflation unemployment) (Targett, 1999)

Time-series methods • Time series methods – predict future values of a variable solely from historical values of itself • Involves determining patterns in the historical record and then projecting the patterns into the future • Time series is preferable when: • Conditions are stable and will continue to be so in the future • Short-term forecasts are required and there is not enough time for the conditions to change more than a little • A base forecast is needed on to which can be built changes in future conditions (Targett, 1999)

Time-series data smoothing • Long run and often short-run data is not stationary (where fluctuations in the data series for a time period relative to another are relatively constant) • To enhance forecasting, without completely repeating time-series baseline efforts, the original time-series data can be built upon with new data • Data smoothing allows for the replacement of the original series by replacing each observation with the average of it and other observations either side of it; if each average is calculated from three actual observations it is said to be a three point moving average; if each average is from five actual observations it is said to be a 3 point moving average (Targett, 1999)

Forecasting applications – based on time horizon • Short-term (up to 1 year) – time series • Medium-term (1-5 years) – causal modeling • Long-term (more than 5 years) – qualitative • Note: data availability and the nature of forecast required also influence the type of technique to be used (Targett, 1999)

Quantitative testing of the accuracy of time-series and causal modeling forecasts • Simple interrupted time-series – the basic structure of the interrupted time-series design entails (1) a series of pre-intervention observations taken on the target population to establish a baseline trend, (2) an intervention at a known point in this sequence, and (3) a series of post-intervention observations from which to infer change from before or after • The purpose of analysis is to see whether the pre- and post-treatment series differ (Rog, 1994) • The interrupted time-series becomes less interpretable as the intervention point is less clear, when knowledge of an intervention or the intervention itself diffuses slowly through a community, when effects are delayed rather than instantaneous (Rog, 1994)

Regression-discontinuity in Evaluation • interventions are often designed so that individual with specific needs can receive special services, or so that those with special merit can receive extra resources • Although many individuals might benefit from these resources, we cannot afford to include everyone; in such cases we are forced to admit only a select few, and a sense of fairness compels us to choose only those who are the neediest • The regression-discontinuity design is appropriate for assessing the consequences of this decision; its efficacy depends primarily on the assignment of individuals to groups based on a fully known quantitative score that defines program eligibility(Rog, 1994)

Defining and identifying the targets of intervention • Definition of targets is critical but is complicated by the fact that target definitions become moving targets, as do their size estimates • Target definition is most critical at two stages: 1. Pre-design phase of laying out policy/program, 2. During the course of designing the specific content and structure of a program (Rossi, Freeman and Lipsey, 1999)

What is a target? • “the targets of social programs are usually individuals but they may also be groups (families, work teams, organizations), geographically and politically related areas (such as communities), or physical units (houses, road systems, factories). Whatever the target, it is imperative at the outset of a needs assessment to define the units in question early”(Rossi, Freeman and Lipsey, 1999)

Indirect targets • Where a target population is effected indirectly through an intermediate population (i.e., train the trainer scenarios) • Program effectiveness then needs to be measured largely on whether the pre-determined pathways, intermediate to final targets, are correctly identified in the program theory (Rossi, Freeman and Lipsey, 1999)

Specifying targets • Need to strategically balance all stakeholder perspectives with issues of scientific rigor • Risk is that targets can be defined to broad or can be overly inclusive • Over-restrictive definitions which are minimally inclusive can also result • Target definitions may be impossible to put into practice if framed incorrectly

Varying perspectives on target specification • “although needs assessment cannot establish which perspective on program targets is correct, it can help eliminate conflicts that might arise from groups talking past each other. This is accomplished by investigating the perspectives of all the significant stakeholders on target definition and helping ensure that none is left out of the decision-making process through which the program focus is determined. Information collected about needs from varying perspectives may lead to a reconceptualization of the target population or of the prospective intervention, or even indicate the advisability of abandoning the program (especially if the different perspectives turn out to be contradictory and intensely held by the various stakeholders)” (Rossi, Freeman and Lipsey, 1999)

Target definition – incidence and prevalence • “incidence refers to the number of new cases of a particular problem that are identified or arise in a specified geographical or otherwise defined area during a specified time period”(Rossi, Freeman and Lipsey, 1999) • “prevalence refers to the number of existing cases in a particular area at a specified time”(Rossi, Freeman and Lipsey, 1999)

Health sector planning – incidence and prevalence • When analyzing or assessing disorders or a short duration (i.e., upper respiratory infection), planners tend to be more interested in incidence • When analyzing or assessing the effects of long-term disorders (i.e., chronic diseases), prevalence is of greater interest to planners

Hybrid uses – incidence and prevalence • Some problems, such as unemployment, meaningfully use incidence or prevalence under different circumstances • “the principle involved centers on the issue of whether one is concerned with detecting and treating new cases as they appear or with existing cases whatever their time of origin”(Rossi, Freeman and Lipsey, 1999)

Population at risk • Refers to “that group of persons or units that has a significant probability of developing a given condition”(Rossi, Freeman and Lipsey, 1999) • Probabilistic terms are the only way in which to describe the population at risk

Sensitivity and Specificity • Sensitivity – program screening that selects properly for true positives • Specificity – ability to exclude all false positives (persons who do not meet all relevant criteria)

Need and demand • “whereas a population at risk includes all those with a high probability of having or acquiring a given condition, a population in need is a group of potential targets who currently manifest the condition”(Rossi, Freeman and Lipsey, 1999) • Demand – refers to a desire or willingness to obtain or receive a service (only partial overlap with criteria associated with need) • “Clearly, the distinctions between populations at risk, in need, and at demand are important for estimating the scope of a problem, anticipating the size of the target population, and subsequently designing, implementing, and evaluating the program”(Rossi, Freeman and Lipsey, 1999)

Rates • Rates allow for the description of proportions of a population with a particular problem • Rates often allow for much easier and more relevant comparisons than do frequency comparisons • “in most cases, it is not only traditional but also useful to specify rates by age and sex. In communities in which there are marked sub-cultural differences, racial, ethnic, and religious groups also become important denominators for the disaggregation of characteristics. Other variables useful in identifying characteristics of the target population include socioeconomic status, geographic location, and residential mobility”(Rossi, Freeman and Lipsey, 1999)

Describing the nature of service needs • Often adaptation to local nuances of the problem are necessary, gleaned through referencing those who experience the problems – including their perceptions on existing or required services and programs and barriers to receiving assistance • Need to here from many perspectives

Qualitative methods for describing needs • Methods should be careful, thoughtful • Focus group- can be a rich information source for defining the problem and the service needs of those who experience it • Focus groups are particularly useful when they bring together knowledgeable persons, are strategically composed, and when they are well facilitated • A useful strategy is to use qualitative methods to obtain information about the nature of the problem (exploratory stage), and quantitative assessment then builds on this information to form better estimation of the extent and distribution of the problem (Rossi, Freeman and Lipsey, 1999)

Factors which lead to low focus group creativity • 1. The group has no common goal or core focus (selection and facilitation) • 2. No standard method for making decisions is followed (preparation and facilitation) • 3. The process of generating ideas is not separated from the evaluation of ideas (facilitation) • 4. Ideas are not evaluated on their own merits. They are evaluated in terms of which group member/s suggested them (facilitation)

5. Ideas do not become the property of the group once they are suggested (preparation and facilitation) • 6. Conclusions are an individual product instead of a group product (facilitation) • 7. The group does not perceive member’s time as a valuable and scarce resource (selection)

8. Some members do not feel sufficiently at ease to participate and submit their ideas (they fear derision or reprisal) (selection and facilitation) • 9. Some members dominate or deflect the group from its stated purposes (selection and facilitation) • 10. The group places pressure on its members to make their behavior conform to a low standard of creativity ( Dailey, Keenan, and Tayeb, 1998)

Guidelines on focus group size • Middle-sized groups (5 to 11 members) tend to make more accurate decisions than groups outside of this range • Small groups (2 to 5 members) are better able to achieve consensus than large groups • Larger groups (11 or more members) generate more ideas, but as size increases beyond 20 members, the number of ideas relative to the number of members decreases • Groups of 4 to 5 members foster greater member satisfaction than middle-sized or large groups • Very small groups (2 to 3 members) can make members very anxious about their high performance visibility (Dailey, Keenan and Tayeb, 1998)

Exhibit 4-L: Sample Protocol for a needs assessment focus group • Protocol – list of topics and open ended questions to guide the group process • The protocol needs to: • Ask questions in a logically sequential order • Be open-ended and topical to avoid leading and to stimulate creativity • Carve out manageable topic chunks

Example focus process • Introduction – greetings; explanation of purpose; introductions; ground rules; how the focus group works; and, how results will be used • Introduce topic chunk (e.g. barriers to service); ask for member views on barriers; probe further based on your understanding of typical barrier issues (i.e., waiting lists, transportation, inclusivity) • Probe regarding prioritization of barriers • Ask for solutions • Debrief and wrap-up (facilitator summarizes, clarifies, and addresses questions)(Rossi, Freeman and Lipsey, 1999)

CRJS 4466 PROGRAM & POLICY EVALUATION LECTURE #4