Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good

Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good • Identification and formulation of questions is a critical phase of the evaluation process • The essence of evaluation is generating credible answers to questions about the performance if a social program. • Questions must be answerable with the resources available

Evaluation revolves around the hub of carefully crafted evaluation questions • Questions need to be meaningful for stakeholders and program decision makers • Who will use the the evaluation results? • What info do they need? • How do they expect to use it? • The evaluator’s own analysis of the program is also important

Communication with decision makers and stakeholders • Questions are formulated through discourse and negotiation • Engaging these groups increases their understanding and appreciation, and makes effective use of the findings when available • Evaluators can’t depend solely on their perspectives – they are not experts on evaluation

Role of the Evaluator • Framing the questions – the evaluator knows how to analyze a program and focus an evaluation. • Raise issues that might be overlooked • Identify aspects of program’s operation and outcomes that warrant inquiry • Draw out concerns from stakeholders that can be translated into questions that can be answered by evaluation research

Written summary of questions Helps evaluator guide the evaluation design. This is a useful reference for design and selecting research procedures. Can be shared with stakeholders to ensure their concerns are addressed. Safeguards against later misunderstanding of what the evaluation was suppose to accomplish.

The 2 most important topics related to evaluation questions: • 1. How to formulate evaluation question that can be addressed using research procedures available to the evaluator • 2. How to determine specific questions that the evaluation should focus.

What makes a good evaluation question? • Focus the evaluation on the areas of program performance at issue for key decision makers and stakeholders • Facilitate design of data collection to provide meaningful info about performance • Identify a distinct dimension of program performance that can be credibly assessed

What it means to evaluate something • Establishing criteria of merit – on what dimension must the evaluand do well? • Constructing standards – how well should should the evaluand perform? • Measuring performance and comparing with standards – How well did the evaluand perform? • Synthesizing and integrating data into a judgment of merit or worth. What is the merit or worth of the evaluand

……a thought • To evaluate anything means to assess the merit or worth of something against criteria and standards.

Dimensions of Program Performance • Good evaluation questions must first be reasonable and appropriate • The questions must be answerable – specific, concrete, practical, and measurable

Reasonable and Appropriate Questions • Work with stakeholders to scale down and focus the questions – • Lit Review – social science and social service literature • Get acquainted with the program – its structure, activities, roles and tasks of personnel, nature of participants, and assumptions.

Questions must be Answerable • Easy to formulate unanswerable questions without realizing it….Does this program enhance family values? • Must be possible to identify some evidence or “observables” that can be obtained and will be credible as the basis for an answer. • Have measurable performance dimensions stated in unambiguous terms with non-controversial definitions.

Criteria for Program Performance • What distinguishes evaluation questions is they have to do with performance and are associated with criteria by which performance can be judged. • Performance Criterion (or standard) that apply, as well as the performance dimension at issue. • An evaluation that only describes performance, but doesn’t assess it, is not truly evaluation.

Standards by which program performance may be evaluated: • Needs/wants of target population • Program goals and objectives • Professional standards • Customary practice, norms • Legal requirements • Ethical/moral values: social justice, equity

Standards…continued • Past performance; historical data • Targets set by program managers • Expert opinion • Pre-intervention baseline levels for target population • Conditions expected in the absence of the program • Cost or relative cost

Places to look for performance criteria: • Professional standards – esp in medical and health programs • Prior experience • Evaluation and program literature • Judgement ratings from stakeholders to establish criterion levels or ranges (low to high performance)

Typical evaluation Questions • Need for program services (needs assessment) • Program’s conceptulization or design (assessment of program theory) • Program operations and service delivery (assessment of program process) • Program outcomes (impact assessment) • Program cost and efficiency (efficiency assessment)

Order of the five categories or questions • The questions draw meaning from the answers to the prior questions. • Implementation failure – poor outcomes due to program activities needed to bring about desired improvements did not actually occur. Nutritional status of homeless people did not improve because soup kitchen was rarely open. • Theory failure- Program conceptualization and design can’t generate desired outcomes no matter how well implemented – soup kitchen was far away.

Evaluation Hierarchy – Assessment of: • 1. Need for the Program • 2. Program Design and Theory • 3. Program Process and Implementation • 4. Program Outcome?Impact • 5. Assessment of Program Cost and Efficiency

When developing questions • Start at the bottom of the evaluation hierarchy and consider what is known and what needs to be known about the issues • Then move up the ladder • Logical interdependencies between the levels • Premature attention to higher-order evaluation questions can be avoided

Specific Questions Evaluators Should Ask • Don’t rely only on input from sponsors or stakeholders – they are familiar with the program and can overlook critical, routine aspects or performance. • Evaluator should make an independent analysis of program performance pertinent for investigation

continued • Multiple stakeholders = multiple views • Set priorities with them and integrate as many relevant concerns as possible • If they have constraints, the evaluator must decide whether or not to conduct the evaluation anyway or negotiate to broaden additional perspectives

Essential • Strive to ensure stakeholders understand and accept the nature of the eval process, the type of info it will produce, what it might mean if results come out one way or another, and what ambiguities or unanswered questions remain.

Snowball approach • Identify and contact a person or group, they then nominate others, who in turn nominate others. • The process ends when no new nominations are given.

Topics to discuss with Stakeholders • Why the evaluation is needed • The program goals and objectives • What are the most important questions for the evaluation to answer?

Backward mapping Start with a specification of the desired endpoint and work backward to determine what must be done to get there.

Analysis of Program Assumptions and Theory • Construct a conceptual model of how the program is expected to work and the connections presumed between activities and functions and social benefits it is intended to produce. • Program theory is the set of assumptions about relationships between strategy and tactics and the social benefits it expects to produce. • Program theory is also called program conceptulaization, program plan, blueprint or design.

Logic Model • Lays out expected sequence of steps going from program services to client outcomes.

Collating Eval Questions and Setting Priorities • Questions tend to cluster around program functions – ex services or outcomes • Also around issues – ex need, design, impact • When the evaluator prioritizes the questions for a selected program, the next phase is design.

Summary • Assess the need for a program • Program theory or plan for addressing the need • Implementation of the program and associated process • Impact or outcome of the program implementation on the social need • Efficiency with which the program attains its outcomes

Chapter 3 Identifying Issues and Formulating Questions – Mary Ellen Good