The schedule moving forward…

1. The schedule moving forward… • Today • Evaluation Research • Remaining time = start on quantitative data analysis • Thursday • Methodology section due • Quantitative analysis continued (start Exercise 10) • Next Tuesday • Exercise 10 Due • Maahs returns Exercise #9 • Research proposal (everything is now online) is due on Dec 21st (Monday)

2. Evaluation Research • PURPOSE rather than specific method • Much more popular in last 20 years • A form of “applied” research • Results, though designed to impact decision-making, often have no impact

3. Types of Evaluation Research • Needs assessment • Cost-benefit study • Monitoring study • Program evaluation

4. Doing evaluation (program) research • Research questions / measurement as Job #1 • Political context how do measure • Operationalize “outcome” of interest • “Response” variable • NEED AGREEMENT: what is the program trying to do? • Operationalize “processes” • Intermediate objectives • Context of • ”

5. Research designs for evaluation • Experimental • Problems, ethics (placebo) • The “black box” issue • Quasi-experimental designs • Time Series (multiple time series) • Qualitative evaluations • Low birth weight study

6. Evaluation Research ISSUES • Evaluation research as “MESSY” • Administrative control, context of “real life” • Importance of looking at “black box” • Ethics • The intervention itself can raise issues • Control group members • Violating Ethics 101 (Tuskegee) • Evaluation impacts people’s lives

7. The (non) use of findings • Purely rational/scientific world vs. the one we inhabit • Nixon panel on porn • Scared Straight • 3 Strikes and You’re Out legislation • Studies gone wrong… • Fire the evaluator (or dismiss as pointy headed idiot) and keep the program • Why not trust/use the results? • Scientist/practitioner gap • “True believers” • Vested interests

8. Bivariate Analysis Backed up with a little inferential statistics……yeah baby!

9. Review • Descriptive statistics • Purpose? • Types? • These are “univariate” statistics • Explanatory research • Attempt to demonstrate cause-effect • Necessary Criteria?

10. Demonstrating Associations • Bivariate (2 variables) analysis • There are a number of ways to do this • The method you choose depends largely on characteristics of the two variables • Levels of Measurement?

11. Contingency Tables (cross tabs) • Some find these very intuitive…others struggle • It is very easy to misinterpret these critters • Convention: the independent variable is on the top of the table (dictates columns) and the dependent variable is on the side (dictates rows). • What is in the individual “cells?” • Frequencies (number of cases that fit criteria) • Convert to Percentages: a way to standardize cells and make relationships more apparent

12. Example • A survey of 10,000 U.S. residents • Research question: is one’s political view related to attitudes towards police? • What is the DV and IV? • In constructing a table, what goes where?

13. An Example

14. The Percentages of Interest

15. Inferential Statistics • Researchers are typically not interested in whether there is a relationship in the sample • Want to know about the population • Why might there be a difference between what you find in the sample and what actually exists in the pop? • Even with a probability sample, there is always sampling error • Without a probability sample = error + bias

16. Inferential Statistics II • Cannot assume that the relationship in your sample is true in population • BUT: probability theory allows us to estimate: “The likelihood of obtaining a particular finding if in the population, there was no relationship” OR “The likelihood of obtaining a particular finding assuming the null hypothesis”

17. Test Statistics and Significance Tests • What does “statistically significant” mean? • Not due to sampling error • What do you need to do to test for statistical significance? • A “test statistic” • An indicator of how different the sample is from “no relationship” • The level of error that is acceptable (.05, .01) • Degrees of Freedom

18. The Test Statistic for Contingency Tables • Chi Square, or χ2 • Calculation • Observed frequencies • Expected frequencies • Can get confusing calculating these critters by hand • Intuitive: how different are the observed cell frequencies from the expected (under null hypothesis) cell frequencies • Degrees of Freedom: • (# of Rows -1) (# of Columns -1)

20. The Chi-Square Sampling Distribution (Assuming Null is True)

21. Conventional Significance Testing • Calculate the test statistic • Set your “Type II” error level • The risk of being wrong that you are willing to live with • Find the “critical region” within the distribution for your sample statistic • How far out on the curve does your test statistic have to get before you can reject the null hypothesis • Decide whether to reject, or fail to reject the null hypothesis

22. Significance Testing in SPSS • Within your crosstabs, select “Chi-Square” • SPSS gives you the χ2 value • What you would calculate based on the observed and expected cell frequencies • SPSS also gives you “p” • The exact probability of obtaining this χ2, if indeed the null hypothesis was correct • In newer versions of SPSS, the p values are listed under “sig” or “significance” or something similar

23. Review of Contingency Tables • Level of Measurement • Both IV and DV are Nominal or Ordinal (Categorical data) • Constructing a table • IV on top (columns), DV on bottom (rows) • With this format, select “column percentages”

24. Review of Inferential Statistics • Purpose of inferential statistics • Figure out the odds that a finding from sample is due to “chance” or “sampling error” • Process • Assume null hypothesis is correct • Calculate the odds of obtaining your particular finding under this assumption • If the odds are very low, you may be suspicious that the null hypothesis is incorrect • At some point (research sets level), you say that the odds are so low that you are going to go ahead and reject the null hypothesis

25. Review of Chi-Square • A measure of how different observed (from your sample data) cell frequencies are from what would be expected under the null hypothesis (e.g., no relationship) • The Chi-square distribution changes shape with different degrees of freedom • This is because, by definition, the more cells you have, the higher χ2 gets • df = (R-1)(C-1)

26. Interpreting Chi-Square • Chi-square has no intuitive meaning, it can range from zero to very large • The real interest is the “p value” associated with the calculated chi-square value • This is the exact probability of obtaining the chi-square if in the population there was no relationship • In other words, the exact probability of finding that chi-square, under the null hypothesis