Assessing the need for a program

1. Assessing the need for a program CRJS 4466EA Policy and Program Evaluation

2. Introduction • Needs assessment general definition – “a systematic approach to identifying social problems, determining their extent, and accurately defining the target population to be served and the nature of their service needs” (Rossi, Freeman and Lipsey, 1999)

3. What is a need? • A need is “a social construction negotiated between a set of social agents with responsibility for social programs and policy and a set of claimants and their advocates who assert that a problem exist that warrants intervention” (Rossi, Freeman and Lipsey, 1999) • Within the context of program evaluation, the “primary focus of needs assessment is not on human needs broadly defined but, rather, on social conditions deemed unsatisfactory through some process of social judgement and presumed remedial by social programs” (Rossi, Freeman and Lipsey, 1999)

4. Roles in needs assessment • Politicians, advocates and moral leaders provide the impetus for policy and program change – “shoot from the hip” • Evaluators contribute in a mundane way by providing diagnostic advise • The policy and program arena is replete with ill-effects examples of “shooting from the hip” and the absence of appropriate diagnostics • 1960’s urban planners supported demolition of dilapidated buildings and relocation of poor to replacement housing • A Midwestern American suburb extrapolated federal adolescent crime statistics to their own community • Gun control has focused on legislative remedies forbidding the sale of guns to persons convicted of felony offences (Rossi, Freeman and Lipsey, 1999)

5. Two types of needs assessment • 1. Assessment of need for a specific program or program proposal • 2. As a planning and decision-making tool for policymakers to determine how resources should be allocated across various service areas

6. Defining social problems • As indicated earlier, social problems are social constructions; social problems are not objective phenomena (Rossi, Freeman and Lipsey, 1999) • Program docs and funding proposals generally expose the social problem of the program – minimally these sources can lead to a more probing assessment of problem definition and what other perspectives might be applicable

7. Specifying the extent of the problem • Much more difficult to develop good estimates of problem extent than to identify and define a problem • Many social problems are largely invisible obscuring good estimation (e.g. population of abused children and characteristics of their perpetrators) • Multi-methods/ sources are necessary to estimate otherwise invisible pockets • Overestimation generally results from single-sourcing (e.g. estimation of rate of spousal abuse during pregnancy based on women in shelters) (Rossi, Freeman and Lipsey, 1999)

8. Using existing data sources for estimation • Some social issues can be estimated with confidence using routinely collected census data • When specific census data is not available, indirect estimates can sometimes be made if empirical relationships between available information and problem indicators are known • Undercounts are not uncommon on several variables with census data (e.g. counts of numerous ethnic groups, homeless) • Sometimes undercounts are politically unattractive to correct technically (resource implications; revision of electoral boundaries – U.S.) • In absence of good census data, and where validity of data is not well recognized, consider substantially agreed upon findings from several surveys (Rossi, Freeman and Lipsey, 1999)

9. Using social indicators to identify trends • On some topics data is collected periodically which can be used for trend analysis (e.g. population census information on household incomes, family size and sex) • This data, also known as social indicators, can be used to: • Estimate the size and distribution of a social problem • Alert decision-makers to whether specific social conditions are improving, worsening or stable • In a crude way, estimate the effects of social programs

10. “unfortunately, the social indicators currently available are limited in their coverage of social problems, focusing mainly on issues of poverty and employment, criminal victimization, national program participation, and household composition” (Rossi, Freeman and Lipsey, 1999)

11. Estimating problem parameters through social research • In many instances, no existing data source will provide estimates of the extent and distribution of a problem of interest • Or, good information may be available for a national or regional sample that can not be dissaggregated to a local level • The evaluator must then collect new data whilst being mindful of funds available and precision required (Rossi, Freeman and Lipsey, 1999)

12. Sources of problem parameter information • Agency records – the search may end here if the records are fastidiously kept although such is not usually the case; there also may be some danger in extrapolating to broader populations from a micro-population within an agency • Surveys and censuses-where no credible data exists original research may be required in the way of surveys or even complete enumerations which can both be quite resource intensive; costs can be reduced through telephone • Key informant surveys –surveying of persons whose position or experience should give them some perspective on the magnitude and distribution of the problem; validity is usually poor though; this approach is better used for info collection about population characteristics and needs but not of numbers of persons affected (Rossi, Freeman and Lipsey, 1999)

13. Forecasting need • Forecasting future trends can be quite risky, which is not to say that they should not be used, but that they should be reviewed critically • Forecasting techniques can be qualitative, causal modeling or time-series

14. Forecasting techniques • Qualitative techniques – based on judgement rather than records of past data; better track record of prediction when they are conducted thoughtfully and in a systematic manner (i.e., nominal groups, delphi groups, relevance trees) • Causal modeling – variable to be forecast is related statistically to one or more other variables which are thought to cause changes in it; nearly always this means that regressions analysis in conjunction with past data is used to establish the relationship (e.g. policies such as restricting the money supply are fed into the model to give forecasts of inflation unemployment) (Targett, 1999)

15. Time-series methods • Time series methods – predict future values of a variable solely from historical values of itself • Involves determining patterns in the historical record and then projecting the patterns into the future • Time series is preferable when: • Conditions are stable and will continue to be so in the future • Short-term forecasts are required and there is not enough time for the conditions to change more than a little • A base forecast is needed on to which can be built changes in future conditions (Targett, 1999)

16. Time-series data smoothing • Long run and often short-run data is not stationary (where fluctuations in the data series for a time period relative to another are relatively constant) • To enhance forecasting, without completely repeating time-series baseline efforts, the original time-series data can be built upon with new data • Data smoothing allows for the replacement of the original series by replacing each observation with the average of it and other observations either side of it; if each average is calculated from three actual observations it is said to be a three point moving average; if each average is from five actual observations it is said to be a 3 point moving average (Targett, 1999)

17. Forecasting applications – based on time horizon • Short-term (up to 1 year) – time series • Medium-term (1-5 years) – causal modeling • Long-term (more than 5 years) – qualitative • Note: data availability and the nature of forecast required also influence the type of technique to be used (Targett, 1999)

18. Quantitative testing of the accuracy of time-series and causal modeling forecasts • Simple interrupted time-series – the basic structure of the interrupted time-series design entails (1) a series of pre-intervention observations taken on the target population to establish a baseline trend, (2) an intervention at a known point in this sequence, and (3) a series of post-intervention observations from which to infer change from before or after • The purpose of analysis is to see whether the pre- and post-treatment series differ (Rog, 1994) • The interrupted time-series becomes less interpretable as the intervention point is less clear, when knowledge of an intervention or the intervention itself diffuses slowly through a community, when effects are delayed rather than instantaneous (Rog, 1994)

19. Regression-discontinuity in Evaluation • interventions are often designed so that individual with specific needs can receive special services, or so that those with special merit can receive extra resources • Although many individuals might benefit from these resources, we cannot afford to include everyone; in such cases we are forced to admit only a select few, and a sense of fairness compels us to choose only those who are the neediest • The regression-discontinuity design is appropriate for assessing the consequences of this decision; its efficacy depends primarily on the assignment of individuals to groups based on a fully known quantitative score that defines program eligibility(Rog, 1994)

20. Defining and identifying the targets of intervention • Definition of targets is critical but is complicated by the fact that target definitions become moving targets, as do their size estimates • Target definition is most critical at two stages: 1. Pre-design phase of laying out policy/program, 2. During the course of designing the specific content and structure of a program (Rossi, Freeman and Lipsey, 1999)

21. What is a target? • “the targets of social programs are usually individuals but they may also be groups (families, work teams, organizations), geographically and politically related areas (such as communities), or physical units (houses, road systems, factories). Whatever the target, it is imperative at the outset of a needs assessment to define the units in question early”(Rossi, Freeman and Lipsey, 1999)

22. Indirect targets • Where a target population is effected indirectly through an intermediate population (i.e., train the trainer scenarios) • Program effectiveness then needs to be measured largely on whether the pre-determined pathways, intermediate to final targets, are correctly identified in the program theory (Rossi, Freeman and Lipsey, 1999)

23. Specifying targets • Need to strategically balance all stakeholder perspectives with issues of scientific rigor • Risk is that targets can be defined to broad or can be overly inclusive • Over-restrictive definitions which are minimally inclusive can also result • Target definitions may be impossible to put into practice if framed incorrectly

24. Varying perspectives on target specification • “although needs assessment cannot establish which perspective on program targets is correct, it can help eliminate conflicts that might arise from groups talking past each other. This is accomplished by investigating the perspectives of all the significant stakeholders on target definition and helping ensure that none is left out of the decision-making process through which the program focus is determined. Information collected about needs from varying perspectives may lead to a reconceptualization of the target population or of the prospective intervention, or even indicate the advisability of abandoning the program (especially if the different perspectives turn out to be contradictory and intensely held by the various stakeholders)” (Rossi, Freeman and Lipsey, 1999)

25. Target definition – incidence and prevalence • “incidence refers to the number of new cases of a particular problem that are identified or arise in a specified geographical or otherwise defined area during a specified time period”(Rossi, Freeman and Lipsey, 1999) • “prevalence refers to the number of existing cases in a particular area at a specified time”(Rossi, Freeman and Lipsey, 1999)

26. Health sector planning – incidence and prevalence • When analyzing or assessing disorders or a short duration (i.e., upper respiratory infection), planners tend to be more interested in incidence • When analyzing or assessing the effects of long-term disorders (i.e., chronic diseases), prevalence is of greater interest to planners

27. Hybrid uses – incidence and prevalence • Some problems, such as unemployment, meaningfully use incidence or prevalence under different circumstances • “the principle involved centers on the issue of whether one is concerned with detecting and treating new cases as they appear or with existing cases whatever their time of origin”(Rossi, Freeman and Lipsey, 1999)

28. Population at risk • Refers to “that group of persons or units that has a significant probability of developing a given condition”(Rossi, Freeman and Lipsey, 1999) • Probabilistic terms are the only way in which to describe the population at risk

29. Sensitivity and Specificity • Sensitivity – program screening that selects properly for true positives • Specificity – ability to exclude all false positives (persons who do not meet all relevant criteria)

30. Need and demand • “whereas a population at risk includes all those with a high probability of having or acquiring a given condition, a population in need is a group of potential targets who currently manifest the condition”(Rossi, Freeman and Lipsey, 1999) • Demand – refers to a desire or willingness to obtain or receive a service (only partial overlap with criteria associated with need) • “Clearly, the distinctions between populations at risk, in need, and at demand are important for estimating the scope of a problem, anticipating the size of the target population, and subsequently designing, implementing, and evaluating the program”(Rossi, Freeman and Lipsey, 1999)

31. Rates • Rates allow for the description of proportions of a population with a particular problem • Rates often allow for much easier and more relevant comparisons than do frequency comparisons • “in most cases, it is not only traditional but also useful to specify rates by age and sex. In communities in which there are marked sub-cultural differences, racial, ethnic, and religious groups also become important denominators for the disaggregation of characteristics. Other variables useful in identifying characteristics of the target population include socioeconomic status, geographic location, and residential mobility”(Rossi, Freeman and Lipsey, 1999)

32. Describing the nature of service needs • Often adaptation to local nuances of the problem are necessary, gleaned through referencing those who experience the problems – including their perceptions on existing or required services and programs and barriers to receiving assistance • Need to here from many perspectives

33. Qualitative methods for describing needs • Methods should be careful, thoughtful • Focus group- can be a rich information source for defining the problem and the service needs of those who experience it • Focus groups are particularly useful when they bring together knowledgeable persons, are strategically composed, and when they are well facilitated • A useful strategy is to use qualitative methods to obtain information about the nature of the problem (exploratory stage), and quantitative assessment then builds on this information to form better estimation of the extent and distribution of the problem (Rossi, Freeman and Lipsey, 1999)

34. Factors which lead to low focus group creativity • 1. The group has no common goal or core focus (selection and facilitation) • 2. No standard method for making decisions is followed (preparation and facilitation) • 3. The process of generating ideas is not separated from the evaluation of ideas (facilitation) • 4. Ideas are not evaluated on their own merits. They are evaluated in terms of which group member/s suggested them (facilitation)

35. 5. Ideas do not become the property of the group once they are suggested (preparation and facilitation) • 6. Conclusions are an individual product instead of a group product (facilitation) • 7. The group does not perceive member’s time as a valuable and scarce resource (selection)

36. 8. Some members do not feel sufficiently at ease to participate and submit their ideas (they fear derision or reprisal) (selection and facilitation) • 9. Some members dominate or deflect the group from its stated purposes (selection and facilitation) • 10. The group places pressure on its members to make their behavior conform to a low standard of creativity ( Dailey, Keenan, and Tayeb, 1998)

37. Guidelines on focus group size • Middle-sized groups (5 to 11 members) tend to make more accurate decisions than groups outside of this range • Small groups (2 to 5 members) are better able to achieve consensus than large groups • Larger groups (11 or more members) generate more ideas, but as size increases beyond 20 members, the number of ideas relative to the number of members decreases • Groups of 4 to 5 members foster greater member satisfaction than middle-sized or large groups • Very small groups (2 to 3 members) can make members very anxious about their high performance visibility (Dailey, Keenan and Tayeb, 1998)

38. Exhibit 4-L: Sample Protocol for a needs assessment focus group • Protocol – list of topics and open ended questions to guide the group process • The protocol needs to: • Ask questions in a logically sequential order • Be open-ended and topical to avoid leading and to stimulate creativity • Carve out manageable topic chunks

39. Example focus process • Introduction – greetings; explanation of purpose; introductions; ground rules; how the focus group works; and, how results will be used • Introduce topic chunk (e.g. barriers to service); ask for member views on barriers; probe further based on your understanding of typical barrier issues (i.e., waiting lists, transportation, inclusivity) • Probe regarding prioritization of barriers • Ask for solutions • Debrief and wrap-up (facilitator summarizes, clarifies, and addresses questions)(Rossi, Freeman and Lipsey, 1999)

40. Nominal group technique (NGT) • This method structures the work group’s creative process to minimize verbal interaction among members; the group which is usually no larger than 9, follows a highly structured procedure: • 1. The question under study is posted in front of the group whose members silently generate ideas in writing without looking at the work of others or discussing the questions (5-10 minutes) • 2. The leader goes around the table and asks each participant to read one idea from his or her notes. The idea is recorded in some way and in round robin fashion, all participants present their ideas for recording until all are shown

41. 3. Each idea recorded is discussed in the order it appeared. The leader reads each item and asks the participants if there are any questions or points needing clarification • 4. Each member records the ideas on paper and rank orders them secretly from 1 to n. The mean average rankings are used as a basis for the group’s decision. The NGT process can end here or the decision may be refined through discussion and rewording.

42. 5. The voting patterns can be analyzed and reasons can be examined to see if more accurate decisions can be made • 6. A final vote is taken in the same way as in step 4. This vote closes the decision loop so members experience closure to the NGT process

43. NGT advantages • Idea generation is separated from idea evaluation • Balanced participation occurs and the effects of dominant individuals are lessened • It ensures the systematic movement towards the aggregation of votes to determine a preferred outcome • A group in any stage of development can use NGT • It is time-efficient and members find the process to be satisfying (Dailey, Keenan and Tayeb, 1998)

44. Delphi technique • This method can be used when members cannot attend a meeting • It is a method for gathering systematically written judgments from members using a set of sequentially modified questionnaires interspersed with summaries of results from previous rounds of information gathering from members • In using it, members of the group may not know the other group members and it requires a central coordinating mechanism to manage the alteration, transmission and summarizing of questionnaire data

45. Delphi steps • 1. Each group member independently and anonymously records comments, suggestions and solutions to the problem facing the group • 2. All the data generated in step 1 are sent to a centrally located individual who is responsible for data compilation and reproduction • 3. Each member receives a copy of all written comments from other members • 4. Members generate feedback on other members’ comments, and all second-round feedback is written down and sent to the centrally-located individual • 5. Steps 3 and 4 are repeated as many times as necessary to reach consensus

46. Delphi group effectiveness • Evidence supports Delphi and NGT approaches over less disciplined procedures • It has been demonstrated that average performance was higher for Delphi and NGT groups than undisciplined groups both working on a comparable problem-solving task • Delphi groups reduce dominant personality effects and perceived member status on group decision-making • Computer based e-mail and world wide web create efficiencies (Dailey, Keenan, and Tayeb, 1998)

47. Delphi/NGT differences • Nominal groups are composed of members who know each other; are designed to meet face to face; are designed to reach a decision more quickly; and, are more likely to have status and dominance effects • Delphi groups are composed of members who are unaware of other group members; made up of members who never meet face to face; and, are reliant on a central processing unit which collects, collates and feeds back information (Dailey, Keenan, and Tayeb, 1998)