PPA 502 – Program Evaluation

1. PPA 502 – Program Evaluation Lecture 4a – Qualitative Data Collection

2. Trained Observer Ratings • A systematic technique for assigning accurate, reliable grades based on direct visual observations. • Trained observers are persons who make ratings of conditions by comparing their perception of the condition to a prespecified rating scale.

3. Trained Observer Ratings • Ratings should contain written definitions or photographic benchmarks. • Systematic procedures to ensure the accuracy and consistency of ratings between raters and over time. • Most applications have focused on facilities maintenance.

4. Trained Observer Ratings • Advantages • Easy and often inexpensive way to quantify conditions and program outcomes that are otherwise difficult to measure. • Despite subjectivity, if done properly, they can introduce considerable objectivity and reliability. • Readily understood by public administrators. • Tend to focus on conditions as experienced by average citizen. • In most cases, ratings can be made by ordinary citizens.

5. Trained Observer Ratings • Disadvantages. • Can only assess characteristics that can be directly sensed. • Can be affected by conditions that limit observability. • Can impose some physical dangers. • Most measures are nominal or ordinal limiting statistical applications. • Not useful for assessing subtle conditions or complex changes. • Without considerable care, imprecision, poor repeatability, and lack of comparability can result. • Credibility can be affected if rater is also implementer. • Ratings can become intrusive.

6. Trained Observer Ratings • Steps. • Selecting characteristic to be rated. • Developing rating scales and forms (single versus multiple.) • Selecting the raters and conducting the ratings. • Quality control procedures. • Analyzing and presenting the results. • Time, cost, and staffing requirements.

7. Expert Judgment • Decision-makers and program evaluators should consider using expert judgment when programs are subject to high uncertainty. • Uncertain inputs. • Uncertain outcomes. • Uncertain causality. • Expert judgments should be reliable (reproducible) and valid.

8. Expert Judgment • Inside expert evaluations. • Managers and administrators. • Program evaluation staff. • Steps • Collecting and collating information. • Checking for validity and reliability. • Implementing recommendations.

9. Expert Judgment • Using outside experts. • Selecting the experts. • Eliciting systematic judgments. • Unstructured, direct interaction (informed dialogue) procedure. • Structured, indirect interactions. • Delphi.

10. Expert Judgment • Delphi steps. • Develop the evaluation issues and questions that the experts will address. • Obtain data that the decision maker wants experts to examine and arrange any desired interviews with program staff. • Design the instrument for addressing the issues and questions. • Select and contact the experts. • Administer the instrument, round #1. • Collate, aggregate, and send the judgments from round #1 back to the experts. This step is done by the person designated to be in control.

11. Expert Judgment • Delphi steps (contd.) • Administer the instrument, round #2. • Repeat step #6. • Administer the instrument, round #3. • Repeat step #6. • Prepare final report on results. • Devil’s advocate. • Dialectical inquiry.

12. Focus Groups • Consumer focus of modern public administration. • What are focus groups? • An informal, small-group discussion designed to obtain in-depth qualitative information. Individuals are specifically invited to participate in the discussion. Participants usually have something in common (demographics, service receipt, agency employment).

13. Focus Groups • Focus group discussion are informal. Interaction is encouraged. Conversation led by a moderator whose role it is to foster interaction. Moderator manages discussion to make sure it does not stray too far from the topic of interest. Moderator also follows up on participant’s comments to obtain further details or to introduce new topics to the group.

14. Focus Groups • Focus groups have the following characteristics: • Each group is kept small to encourage interaction among the members. • Each session usually lasts ninety minutes. • The conversation focuses on a restricted number of topics. The actual number varies depending on the objective of the session but is usually no more than three to five related subjects. • The moderator has an agenda that outlines the major topics to be covered. These topics are usually narrowly defined to keep the conversation relevant.

15. Focus Groups • Differences between focus groups and surveys. • Size and selection of sample respondents. • Construction of questions. • How results are handled. • Surveys – aggregation. • Focus groups – disaggregation and detail.

16. Focus Groups • Focus groups most useful in the exploratory stages of research, or when an administrator wants to develop a deeper understanding of a program or service.

17. Focus Groups • Steps. • Participant selection. • What target population or populations do you want information from? • You should not invite different target populations to the same session. • Definition should be careful and specific to the research questions. • But there are tradeoffs which can reduce the potential pool. • Focus group recruiting is done over the phone with screening questionnaires. • Focus group participants are usually compensated for their time.

18. Focus Groups • Steps. • The agenda. • The agenda or moderator’s guide outlines the major topics to be covered. The moderator refers to this document during the discussion to make sure all major topics are covered. • The focus group moderator usually writes his or her own guide. • Structure. • General question to initiate interaction. • Moderator must decide on the spot whether to allow free-wheeling discussion or structured conversation. Should know in advance if order is critical.

19. Focus Groups • Steps. • Communication with moderator. • Moderator must understand needs of sponsor in preparing the agenda. • In-house or professional research. • Facilities. • Usually conducted in special facilities with video- and audio-taping equipment. Ethical considerations paramount here. • Schedules. • Usually on weekday evenings. • Multiple sessions. • Competing activities. • Lead time necessary for preparation. Minimum of three weeks.

20. Focus Groups • Reports. • Debriefing summaries. • Descriptions of the discussion and key points made. Not quantitative. • Pitfalls • Inappropriate to research questions (generalizability). • Inadequate communication between sponsor and moderator. • Sponsor defensiveness.