Download
1 / 26
260 likes | 583 Views
Health Program Effect Evaluation Questions and Data Collection Methods. CHSC 433 Module 5/Chapter 9 L. Michele Issel, PhD UIC School of Public Health. Objectives. Develop appropriate effect evaluation questions List pros and cons for various data collection methods
E N D
Health Program Effect Evaluation Questions and Data Collection Methods CHSC 433 Module 5/Chapter 9 L. Michele Issel, PhD UIC School of Public Health
Objectives • Develop appropriate effect evaluation questions • List pros and cons for various data collection methods • Distinguish between types of variables
Involve Evaluation Users so they can: • Judge the utility of the design • Know strengths and weaknesses of the evaluation • Identify differences in criteria for judging evaluation quality • Learn about methods • Have debated BEFORE have data
Terminology The following terms are used in reference to basically the same set of activities and for the same purpose: • Impact evaluation • Outcome evaluation • Effectiveness evaluation • Summative evaluation
Nature of problem addressed:new knowledge vs assess outcomes Goal of the research: new knowledge for prediction vs social accounting Guiding theory: theory for hypothesis testing vs theory for the problem Appropriate techniques: sampling, statistics, hypothesis testing, etc. vs fit with the problem Differences between Research - Evaluation
Evaluation Questions… • What questions do the stakeholders want answered by the evaluation? • Do the questions link to the impact and outcome objectives? • Do the questions link to the effect theory?
From Effect Theory to Effect Evaluation • Consider the effect theory as source of variables • Consider the effect theory as guidance on design • Consider the effect theory as informing the timing of data collection
From Effect Theory to Variables The next slide is an example of using the the effect theory components to identify possible variables on which to collect evaluation data.
Impact vs Outcome Evaluations • Impact is more realistic because it focuses on the immediate effects and participants are probably more accessible. • Outcomes is more policy, longitudinal, population based and therefore more difficult and costly. Also, causality (conceptual hypothesis) is fuzzier.
Effect Evaluation Draws upon and uses what is known about how to conduct rigorous research: Design --overall plan, such as experimental, quasi-experimental, longitudinal, qualitative Method -- how collect data, such as telephone survey, interview, observation
Methods --> Data Sources • Observational--> logs, video • Record review--> Client records, patient chart • Survey--> participants/not, family • Interview--> participants/not, • Existing records --> birth & death certificates, police reports
Comparison of Data Collection Methods Characteristics of each method to be considered when choosing a method: • Cost • Amount of training required for data collectors • Completion time • Response rate
Validity and Reliability • Method must use valid indicators/measures • Method must use reliable processes for data collection • Method must use reliable measures
Variables, Indicators, Measures • Variable is the “thing” of interest, variable is how that thing gets measured • Some agencies use “indicator” to mean the number that indicates how well the program is doing • Measure the way that the variable is known It’s all just language…. Stay focused on what is needed.
Types of Effects as documented through Indicators Indicators of physical change Indicators of knowledge change Indicators of psychological change Indicators of behavioral change Indicators of resources change Indicators of social change
Advise It is more productive to focus on a few relevant variables than to go on a wide ranging fishing expedition. Carol Weiss (1972)
Variables • Intervening variable: any variable that forms a link between the independent variable, AND without which the independent variable is not related to the dependent variable (outcome).
Variables • Confounding variable is an extraneous variable which accounts for all or part of the effects on the dependent variable (outcome); mask underlying true assumptions. • Must be associated with the dependent variable AND the independent variable.
Confounders • Exogenous (outside of individuals) confounding factors are uncontrollable (selection bias, coverage bias). • Endogenous (within individuals) confounding factors equally important: secular drift in attitudes/knowledge, maturation (children or elderly), seasonality, interfering events that alter individuals.
Variable story… To get from Austin to San Antonio, there is one highway. Between Austin and San Antonio there is one town, San Marcus. San Marcus is the intervening variable because it not possible to get to San Antonio from Austin without going through San Marcus. The freeway is often congested, with construction and heavy traffic. The highway conditions is the confounding variable because it is associated with both the trip (my car, my state of mind) and with arriving (alive) in San Antonio.
