1 / 26

The schedule moving forward…

The schedule moving forward…. Today Evaluation Research Remaining time = start on quantitative data analysis Thursday Methodology section due Quantitative analysis continued (start Exercise 10) Next Tuesday Exercise 10 Due Maahs returns Exercise #9

lana
Download Presentation

The schedule moving forward…

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The schedule moving forward… • Today • Evaluation Research • Remaining time = start on quantitative data analysis • Thursday • Methodology section due • Quantitative analysis continued (start Exercise 10) • Next Tuesday • Exercise 10 Due • Maahs returns Exercise #9 • Research proposal (everything is now online) is due on Dec 21st (Monday)

  2. Evaluation Research • PURPOSE rather than specific method • Much more popular in last 20 years • A form of “applied” research • Results, though designed to impact decision-making, often have no impact

  3. Types of Evaluation Research • Needs assessment • Cost-benefit study • Monitoring study • Program evaluation

  4. Doing evaluation (program) research • Research questions / measurement as Job #1 • Political context how do measure • Operationalize “outcome” of interest • “Response” variable • NEED AGREEMENT: what is the program trying to do? • Operationalize “processes” • Intermediate objectives • Context of • ”

  5. Research designs for evaluation • Experimental • Problems, ethics (placebo) • The “black box” issue • Quasi-experimental designs • Time Series (multiple time series) • Qualitative evaluations • Low birth weight study

  6. Evaluation Research ISSUES • Evaluation research as “MESSY” • Administrative control, context of “real life” • Importance of looking at “black box” • Ethics • The intervention itself can raise issues • Control group members • Violating Ethics 101 (Tuskegee) • Evaluation impacts people’s lives

  7. The (non) use of findings • Purely rational/scientific world vs. the one we inhabit • Nixon panel on porn • Scared Straight • 3 Strikes and You’re Out legislation • Studies gone wrong… • Fire the evaluator (or dismiss as pointy headed idiot) and keep the program • Why not trust/use the results? • Scientist/practitioner gap • “True believers” • Vested interests

  8. Bivariate Analysis Backed up with a little inferential statistics……yeah baby!

  9. Review • Descriptive statistics • Purpose? • Types? • These are “univariate” statistics • Explanatory research • Attempt to demonstrate cause-effect • Necessary Criteria?

  10. Demonstrating Associations • Bivariate (2 variables) analysis • There are a number of ways to do this • The method you choose depends largely on characteristics of the two variables • Levels of Measurement?

  11. Contingency Tables (cross tabs) • Some find these very intuitive…others struggle • It is very easy to misinterpret these critters • Convention: the independent variable is on the top of the table (dictates columns) and the dependent variable is on the side (dictates rows). • What is in the individual “cells?” • Frequencies (number of cases that fit criteria) • Convert to Percentages: a way to standardize cells and make relationships more apparent

  12. Example • A survey of 10,000 U.S. residents • Research question: is one’s political view related to attitudes towards police? • What is the DV and IV? • In constructing a table, what goes where?

  13. An Example

  14. The Percentages of Interest

  15. Inferential Statistics • Researchers are typically not interested in whether there is a relationship in the sample • Want to know about the population • Why might there be a difference between what you find in the sample and what actually exists in the pop? • Even with a probability sample, there is always sampling error • Without a probability sample = error + bias

  16. Inferential Statistics II • Cannot assume that the relationship in your sample is true in population • BUT: probability theory allows us to estimate: “The likelihood of obtaining a particular finding if in the population, there was no relationship” OR “The likelihood of obtaining a particular finding assuming the null hypothesis”

  17. Test Statistics and Significance Tests • What does “statistically significant” mean? • Not due to sampling error • What do you need to do to test for statistical significance? • A “test statistic” • An indicator of how different the sample is from “no relationship” • The level of error that is acceptable (.05, .01) • Degrees of Freedom

  18. The Test Statistic for Contingency Tables • Chi Square, or χ2 • Calculation • Observed frequencies • Expected frequencies • Can get confusing calculating these critters by hand • Intuitive: how different are the observed cell frequencies from the expected (under null hypothesis) cell frequencies • Degrees of Freedom: • (# of Rows -1) (# of Columns -1)

  19. The Chi-Square Sampling Distribution (Assuming Null is True)

  20. Conventional Significance Testing • Calculate the test statistic • Set your “Type II” error level • The risk of being wrong that you are willing to live with • Find the “critical region” within the distribution for your sample statistic • How far out on the curve does your test statistic have to get before you can reject the null hypothesis • Decide whether to reject, or fail to reject the null hypothesis

  21. Significance Testing in SPSS • Within your crosstabs, select “Chi-Square” • SPSS gives you the χ2 value • What you would calculate based on the observed and expected cell frequencies • SPSS also gives you “p” • The exact probability of obtaining this χ2, if indeed the null hypothesis was correct • In newer versions of SPSS, the p values are listed under “sig” or “significance” or something similar

  22. Review of Contingency Tables • Level of Measurement • Both IV and DV are Nominal or Ordinal (Categorical data) • Constructing a table • IV on top (columns), DV on bottom (rows) • With this format, select “column percentages”

  23. Review of Inferential Statistics • Purpose of inferential statistics • Figure out the odds that a finding from sample is due to “chance” or “sampling error” • Process • Assume null hypothesis is correct • Calculate the odds of obtaining your particular finding under this assumption • If the odds are very low, you may be suspicious that the null hypothesis is incorrect • At some point (research sets level), you say that the odds are so low that you are going to go ahead and reject the null hypothesis

  24. Review of Chi-Square • A measure of how different observed (from your sample data) cell frequencies are from what would be expected under the null hypothesis (e.g., no relationship) • The Chi-square distribution changes shape with different degrees of freedom • This is because, by definition, the more cells you have, the higher χ2 gets • df = (R-1)(C-1)

  25. Interpreting Chi-Square • Chi-square has no intuitive meaning, it can range from zero to very large • The real interest is the “p value” associated with the calculated chi-square value • This is the exact probability of obtaining the chi-square if in the population there was no relationship • In other words, the exact probability of finding that chi-square, under the null hypothesis

More Related