How To Evaluate Programs/Interventions?

2. How To Evaluate Programs/Interventions? Step 1 – Identify independent and dependent variables/concepts Independent – Intervention Dependent – Stated objective of program Step 2 – Define these variables/concepts Step 3 – Develop indicators (i.e., questions) to measure these variables/concepts

3. Defining Variables/Concepts Challenges with definitions • Concepts may be multi-dimensional • Concepts can be value laden • Concepts can have both a subjective and objective definition—you must choose one • Only concepts that VARY (are also variables) can be used

4. Examples of Multi-DimensionalConcepts • Social class • Social class has an economic, occupational and education component • Prejudice • Although we often speak of prejudice as one term, there are many different types of prejudice (i.e., racial prejudice, social class prejudice, religious prejudice, etc.). These types comprise different components of prejudice.

5. Examples of Value Laden Concepts • Prejudice – not acceptable to have these types of attitudes • Are you prejudiced? • Child abuse – not socially acceptable to engage in these types of behaviors or admit having been a recipient • Have you abused your child today? • Have you ever been abused? • Sexual assault – negative stigma is often associated with having been raped • Have you ever raped anyone? • Have you ever been raped?

6. Measurement Error Results from Value Laden Concepts • Social desirability • There are socially desirable answers to questions • Example – If you asked the question “Have you ever abused your child?” the socially desirable answer would obviously be no • Acquiescence bias • Positive vs. negative answers • Example – If people are evaluating a program and they feel connected or grateful to those in charge, they will positively evaluate every item • Leading questions • Questions that predispose respondents to answer a certain way • Example - Rape is a heinous crime. Do you agree?

7. Subjective vs. Objective Definitions of Concepts • Social class – objective (as defined by others) • Family income, level of education, employment • Social class – subjective (as perceived by the respondents) • Ask – What is your social class? • Ask – Would you say you are poor?

8. When a Concept is a Variable, It can • Vary across time • Vary across place • Vary across individuals or groups

9. Examples of Concepts that Vary • Across time • Crime rates vary across age • Annual income varies across age/life span • Across place • Crime rates vary across rural vs. urban areas • Median incomes vary across rural vs. urban areas • Across individuals/groups • Crime rates vary across gender • Income varies across gender

10. Examples of Concepts that Do NOT Vary • Across time • Sex – if you are born a male, you will die a male • Eye color – generally you have same eye color all your life • Across place • Impact of frost on plants – generally if it freezes, then plants die no matter where you plant them • Whether or not babies are born with teeth does not vary across place • Likelihood that children will be born speaking Chinese • Across Individuals/Groups • Impact of restful sleep – generally you are healthier if you sleep well - no matter who you are • Natural hair color doesn’t vary across gender • IQ scores don’t vary across gender

11. You should (1) take a concept, (2) define it as a variable, and (3) develop an indicator to measure it

12. Variables can be measured at different levels—These include:

13. You should always measure variables at the most precise level possible

14. The same variable can be measured at different levels—For example…

15. Always measure variables at the highest level possible. For example, whenever possible measure age at the ratio level

16. Now you try *Leadership skills is a multi-dimensional concept, and implies a combination of various skills. It would therefore be wise to measure it using multiple indicators.

17. Age – more detailed example • Categorical/Nominal • Are you a baby boomer? (1) yes (2) no • Ordinal • Which of the following would describe you? • Child • Teenagers • Young adult • Adult • Interval • How old are you? • 10-14 • 15-19 • 20-24 • Ratio • How old were you at your last birthday? ___ years Always measure variable at highest level possible

18. Income – more detailed example • Categorical/Nominal • Are you a poor? (1) yes (2) no • Ordinal • Rank you income relative to other college students • Much below average Average • Below average Somewhat above average • Somewhat below average Above average Much above average • Interval • What was your annual income for 2008? • Less than 10,000 20,000 to 29,999 40,000 to 49,999 • 10,000 to 19,9999 30,000 to 39,999 50,000+ • Ratio • How much money did you make last year? _____ dollars ______ cents Always measure variable at highest level possible

19. Now you try *Leadership skills is a multi-dimensional concept, and implies a combination of various skills. It would therefore be wise to measure it using multiple indicators.

20. Possible Answers

21. Questions or comments? Please contact: Carol Albrecht Assessment Specialist USU Extension 979-777-2421 carol.albrecht@usu.edu