Data Analysis: Exploring Relationships

1. Data Analysis: Exploring Relationships Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.ResearchDemystified.org

2. Relationships • The most interesting data analysis often involves looking at relationships. • Do children who attend a pre-school program have higher academic performance than children who do not? • Do some neighborhoods in the city have less crime than others? • Is there a relationship between the wealth of a neighborhood and satisfaction with city services? • Does sunspot activity affect stock market activity? • Does human activity cause climate change? Dr. G. Johnson, www.ResearchDemystified.org

3. NASA: Charts Temperature Anomalies Over Time Source: http://www.nasa.gov/topics/earth/features/temp-analysis-2009.html Dr. G. Johnson, www.ResearchDemystified.org

4. NASA: Temperature Anomalies Over Time • This is a time series analysis. • The program calculates trends in temperature anomalies -- not absolute temperatures — but changes relative to the average temperature for the same month during the period of 1951-1980. • Find this article at: http://www.nasa.gov/topics/earth/features/temp-analysis-2009.html Dr. G. Johnson, www.ResearchDemystified.org

5. NASA: Temperature Anomalies Over Time • NASA article states: “Climate scientists agree that rising levels of carbon dioxide and other greenhouse gases trap incoming heat near the surface of the Earth and are the key factors causing the rise in temperatures since 1880, but these gases are not the only factors that can impact global temperatures.” Dr. G. Johnson, www.ResearchDemystified.org

6. NASA: Temperature Anomalies Over Time • “There's a contradiction between the results shown here and popular perceptions about climate trends," Hansen said. "In the last decade, global warming has not stopped." Dr. G. Johnson, www.ResearchDemystified.org

7. Tough Questions About Climate Change • To what extent does human activity contribute to increased CO2 levels? • To what extent does increased CO2 levels impact the global climate? • Is there an association? Is there a causal relationship? • Remember: association is necessary but not sufficient proof of a causal relationship. Dr. G. Johnson, www.ResearchDemystified.org

8. Climate Change: Tough Questions • What does science tell us? • Does it present specifics about measurement, models, assumptions and limitations of the research • Does it provide the ranges for their estimates? • What is the political spin? • Who are the interest groups lining up on either side and what is their financial stake in winning the policy argument? Dr. G. Johnson, www.ResearchDemystified.org

9. Tough Questions About Climate Change • Policy Analysis: What are the costs of doing nothing versus taking various policy actions? • Exactly what did they measure—and what specific assumptions did they use to arrive at their costs? • Remember: cost-benefit analysis can include direct monetary costs and benefits, opportunity costs—but also social costs and benefits that are not easily reduced to dollars • Remember: who pays and who gains are not necessarily the same • Environment and debt: we gain, future generations pay Dr. G. Johnson, www.ResearchDemystified.org

10. Current Controversy: Climate Change • My View: sophisticated users should be wary of an advocate for a policy position who uses a message of fear supported by “facts” that are, at best, rough estimates. • None of us can pinpoint exactly how much we will spend on food next year. • How can anyone make exact predictions about when we will reach the point of no return for earth’s survival or state that we will spend $40 trillion on climate change “prevention” by 2099 (as some pundit stated on TV)? Dr. G. Johnson, www.ResearchDemystified.org

11. Current Controversy: Climate Change • Best estimates are inherently open to debate and criticism. • To treat them as solid and indisputable fact is to be seduced by political spin. • In my view, there are many reasons besides climate change to switch to renewable energy and sustainable manufacturing anyway. • Common sense tells us that at some point, we will run out of non-renewable energy even if we can’t predict the exact date • Do no harm is a good ethical value, whether in conducting research or living one’s life. Dr. G. Johnson, www.ResearchDemystified.org

12. Current Controversy: Climate Change • Nor I am convinced the economy will be destroyed by that transition. • Money is fluid: if we move it from oil to solar, the money will still circulate in the economy. New jobs will be created as old jobs disappear, new entrepreneurs will emerge as old ones fade. • It is economic change. • Although businesses and employees in those oil-dependent industries will feel the brunt of that transition. • What is likely harm, and if likely to be great, are there policy approaches to minimize harm during the transition? • Change is a constant reality of life-as is choice Dr. G. Johnson, www.ResearchDemystified.org

13. Two Types of Relationships • Association: two variables that appear to be related but are not causally connected • Associations can identify “risk factors”, they are like a canary in a mineshaft • Risk factors: early alcohol use by youth is associated with illegal drug use but it is an indicator of risk rather than the causal factor • Useful in identifying youth likely to experience difficulties later and who may benefit from early intervention programs • Association is necessary but not sufficient to demonstrate a cause-effect relationship Dr. G. Johnson, www.ResearchDemystified.org

14. Two Types of Relationships • Correlation: two variables appear to be in a linear relationship. Implies a causal relationship but 4 conditions must be met for cause-effect relationship: • Logical theory • Time-order • Co-variation • Elimination of rival explanations Dr. G. Johnson, www.ResearchDemystified.org

15. Review of Basic Concepts: Independent and Dependent Variables • When seeking to determine whether there is a cause-effect relationship between two variables, researchers need to identify the independent and dependent variables • The context plays a role so you should expect the researchers to be clear about their rationale for deciding which one is the independent variable and which is the dependent variable • Their choices should make sense • But sometimes it is hard to determine which is causal: sometimes all we can tell is that there is a relationship Dr. G. Johnson, www.ResearchDemystified.org

16. Example: Happiness and TV • A study reported that a relationship between self-reported levels of happiness and the amount of TV watched per day • Unhappy people were more likely to watch more hours of TV than happy people • Does unhappiness cause people to watch more TV or does watching more TV cause people to experience unhappiness? Dr. G. Johnson, www.ResearchDemystified.org

17. Review of Concepts: Dependent Variable (DV) • Variable the researchers want to explain • The variable expected to change because changes in the independent variable • In program evaluation: the outcome measure Dr. G. Johnson, www.ResearchDemystified.org

18. Review of Concepts: Independent Variable (IV) • Variable which occurred first and which the researchers believe explains a change in the dependent variable • It could be a specific treatment in an experiment • It could be a characteristic (like age, gender, etc) used as a statistical control • In program evaluation: the program Dr. G. Johnson, www.ResearchDemystified.org

19. Data Analysis Techniques: Working With 2 Variables Descriptive statistics • Cross-tabulations (Crosstabs) • Also called contingency tables • Comparison of means Dr. G. Johnson, www.ResearchDemystified.org

20. Data Analysis Toolkit: Crosstabs • Used when working with nominal and ordinal data • Can be used with interval/ratio data that have been categorized into ordinal data Dr. G. Johnson, www.ResearchDemystified.org

21. Using Crosstabs • The question: Are boys more likely to take hands-on classes than girls? • We are testing whether there is a difference based on gender. • Put another way is type of class associated with gender? • The independent variable is gender. • The dependent variable is the types of classes: • Hands-on. • Traditional. Dr. G. Johnson, www.ResearchDemystified.org

22. Setting up the Analysis • The analysis always looks at the different categories of the independent variable (boys and girls) and at what percent distribution in the dependent variable, which is the type of classes. • The percent distribution of the dependent variable always totals 100%. (See Table 13.1) • It shows the percent of boys in each the two classes. • It shows the percent of girls in each of the two classes. Dr. G. Johnson, www.ResearchDemystified.org

23. Crosstab Table 13.1 Hands-on Traditional Classes Classes Boys 45% 55% 100% Girls 35% 65% 100% Dr. G. Johnson, www.ResearchDemystified.org

24. Interpreting the Crosstabs Table Boys are somewhat more likely (45%) to take the hands-on classes as compared to girls (35%). Dr. G. Johnson, www.ResearchDemystified.org

25. Table 13.2 Gender and Attitude About the Death Penalty Dr. G. Johnson, www.ResearchDemystified.org

26. Gender: Attitude About the Death Penalty • Gender is the independent variable • Attitude about the death penalty is the dependent variable • Analyzing the table: • Of the 644 men who responded to the survey question, 80 percent favor the death penalty as compared to 68 percent of the 747 women. Dr. G. Johnson, www.ResearchDemystified.org

27. Does Gender Make a Difference in Attitude About the Death Penalty? • Interpreting crosstab table 13.2 • The short answer: While the majority of both men and women favor the death penalty, a greater percent of the men reported favoring it (80 percent) as compared to women (68 percent). Dr. G. Johnson, www.ResearchDemystified.org

28. Crosstab Tables Takeaway lesson • When testing for a relationship using crosstabs, the independent variable can be placed in either the column or in the row. • Remember: it is essential that the percent distribution of each category of the independent variable total 100%. Dr. G. Johnson, www.ResearchDemystified.org

29. Gail’s Analysis Guidelines • Determine which variable is the independent variable and which is the dependent variable. • The percentages for each category of the independent variable will add to 100%. • The total number of respondents for each category of the independent variable is shown since this is the basis for the percentage calculations. • This can be helpful if you are analyzing survey data and some questions have fewer respondents because a large number took an “exit.” Dr. G. Johnson, www.ResearchDemystified.org

30. Gail’s Analysis Guidelines • Data is presented in percents (or percents and counts but not just the counts). • For survey data: round percentages to the nearest whole number because it makes the data easier to remember and avoids giving a false sense of precision. • Rounding rule: less than .5, round down. If .5 or higher, round up. Dr. G. Johnson, www.ResearchDemystified.org

31. Adding Complexity: “Controlling For Stuff” • Controlling for a Third Variable • Two variables could be associated but are not causal connected. A third factor may be causing both: • Classic example: As ice cream consumption rises, drowning rises. • Does ice cream consumption cause drownings? • No—the hidden variable here is summertime. Both ice cream consumption and downings increase in the summertime. Dr. G. Johnson, www.ResearchDemystified.org

32. Controlling For A Third Variable http://pewsocialtrends.org/pubs/750/new-economics-of-marriage#prc-jump Dr. G. Johnson, www.ResearchDemystified.org

33. Controlling For A Third Variable • This analysis looks at education (whether they have are a college graduate or not). And marriage, controlling for gender • The data, shown as a bar graph, shows the that the percent married in 1970 and 2007 based on whether they had a college degree or not Dr. G. Johnson, www.ResearchDemystified.org

34. Controlling For A Third Variable • There are noticeable declines in marriage, but the declines are sharper for those without a college education • Those without a college degree were less likely to be married in 2007 as compared to 1970. • There was also a decline in percent married for those with college degrees. • This was true for both men and women. Dr. G. Johnson, www.ResearchDemystified.org

35. Discussion • What might explain why those without a college degree experienced a larger decline in marriage? • Are there possible economic implications, and if so, what might they be? Dr. G. Johnson, www.ResearchDemystified.org

36. State Employee Survey: “Controlling For Stuff” • Results from a state-wide survey can hide important information • Perhaps there are differences between the agencies; Some might be better managed than others Dr. G. Johnson, www.ResearchDemystified.org

37. State Employee Survey: “Controlling For Stuff” • Perhaps there are differences in satisfaction with specific management practices based on age or gender? • Perhaps there are differences in views about diversity practices based on gender, race or sexual orientation? • Thinking about your organization: what might explain differences in employee satisfaction? Dr. G. Johnson, www.ResearchDemystified.org

38. Data Analysis Toolkit: Comparison of Means Do men earn more than women? Dependent variable: income Independent variable: gender Mean income Men $37,685 Women $34,566 Dr. G. Johnson, www.ResearchDemystified.org

39. Does Some Other Variable Explain Income Differences? • Is salary inequality based on gender? • Another rival explanation might be education: so we need to control for that. be another variable—say education level—that might really be the factor that explains (is related to, is correlated with) difference in men and women’s income. • We can “control for” a third variable by having the computer to analyze the mean income of men and women for each level of education. Dr. G. Johnson, www.ResearchDemystified.org

40. Controlling for a 3rd Variable Average Salaries for Men and Women, Controlling for Education Low Medium High Education Men $25,000 $35,000 $75,000 Women $25,000 $35,000 $75,000 Dr. G. Johnson, www.ResearchDemystified.org

41. What Happened? • In this fake example, the relationship between gender and income disappears. • It is very clear that education is the big causal factor. Dr. G. Johnson, www.ResearchDemystified.org

42. Real Data: Gender and Grade Level, Controlling for Education • The federal civil service is organized by grade levels, going from 1-15 • The top level—Senior Executives-are grades 16-18. • Grades 9 and above are typically held by people in the professional occupations—including supervisors and managers. Dr. G. Johnson, www.ResearchDemystified.org

43. Real Data: Average Grade Level, Controlling for Education Source: merit systems protection board survey, 1991-1992. Dr. G. Johnson, www.ResearchDemystified.org

44. Interpretation? • As educational levels increase, so do grade levels for both men and women. • One conclusion: education appears to matter. • But it is also true that women’s grade levels are lower than men’s at the same level of education. Dr. G. Johnson, www.ResearchDemystified.org

45. Note: Decimal Places Used • Note: in this analysis, decimal places reveal useful information that would be lost if rounded. • Rounding does not make sense when working with real numbers that have a very limited range (1-18, for example). • When analyzing a limited range of real numbers using means, you want to preserve the decimal places. • The is also true with grade point averages and faculty evaluations (rating scales of 1-5 or 1-10) Dr. G. Johnson, www.ResearchDemystified.org

46. Research Design: Correlation with Statistical Controls • Early I presented a design called “correlation with statistical controls”. This is how it is done • Controlling for other variables is a way to try to eliminate possible rival explanations • Does gender explain differences in salaries or are differences explained by differences in education? • If the dataset has other variables, such as years of experience or breaks in employment, these can be used as control variables to test for rival explanations besides discrimination as the cause of salary differences. Dr. G. Johnson, www.ResearchDemystified.org

47. “Controlling for Stuff”—Part Of Planning • The researchers need to consider possible control variables when they are developing their research design and data collection tools. • If they think age, education, or race might be important, they need to build that into their data collection. • If they think traffic will vary by weather conditions, day of the week or time of the day, they need to collect that data. Dr. G. Johnson, www.ResearchDemystified.org

48. Takeaway Lesson • Always Ask: Has their purported relationship met all the criteria for asserting causality? • Relationships are difficult to measure and it is hard to demonstrate cause-effect relationships • A single variable rarely causes anything in itself—the world that public administrators deal with is more complex • Exercise healthy skepticism whenever anyone asserts they know the single cause of a complex phenomenon Dr. G. Johnson, www.ResearchDemystified.org

49. Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.ResearchDemystified.org