Meta-Analysis

1. Meta-Analysis

2. Correlation vs. causation

3. Correlation and Causation “The 2011 Chicago Pedestrian Crash Analysis identified a strong correlation between community areas with high numbers of pedestrian crashes and community areas with high crime rates.”

4. It’s a strange fact that when an area of Chicago has more street crime, it has more car crashes, and when it has less street crime, it has fewer car crashes. What could explain this strange correlation? Let’s consider 3 explanations.

5. Street Crime I had to look up “street crime” because I didn’t know exactly what it meant. Wikipedia quotes London’s police definition: “Robbery, often called 'mugging', and also includes thefts from victims in the street where property is snatched and the victim is not assaulted.”

6. So, Like This…

7. Street Crime Causes Car Accidents There are lots of ways street crime could be responsible for more car accidents. 1. If criminals are running away, they may be reckless crossing the street. 2. If they drive away in their cars, they may drive dangerously. 3. If they buy cars with stolen money, they may care less about their cars. 4. If there are lots of muggings, people may drive more often to avoid the crime that occurs on the streets.

8. Another Reason One person actually had another theory: 5. Street crime results in more people getting shot (if you rob someone with a gun and they don’t give you their money, you shoot them). This means more ambulances on the roads, which tend to drive fast and dangerously.

9. Car Accidents Cause Streetcrime What about the other causal direction? Can car accidents cause crime? This is less plausible, but there are certainly possibilities. 1. If you need money to fix your damaged car, maybe you turn to crime. 2. If someone you know is killed in a car crash, maybe you get angry at the world and become a criminal.

10. One Additional Possibility I’ve at least heard of crimes that work like this. Someone runs into your car while you’re stopped at a red light or stop sign. You get out to talk with them and they rob you and drive away. So maybe it’s not street crime causes car crashes or car crashes cause street crime, but car crashes = street crime!

11. Common Cause Explanations There are also various common cause explanations, where some variable C causes both car crashes and street crime. This seems reasonable. What could C be?

12. 1. “Deviant Tendency” According to this explanation, some people have a “deviant tendency,” which is just a fancy way of saying that they like to break the rules, and they aren’t motivated by the same moral concerns as the rest of us. This deviant tendency, so the explanation goes, causes them to break the rules of the road, which brings about accidents, and to break the rules that prohibit robbery.

13. 2. Poverty Not having money, or medical care, or food, or whatever can obviously lead people to become criminals who rob other people. But can poverty cause car crashes? Here are two ways in which this might happen: by causing people to be lack empathy, and by causing people to be fatalists.

14. Lack of Empathy Empathy is an emotional connection that we have to other people and animals. It is when other people’s happiness makes us happy and other people’s sadness makes us sad. Some evidence suggests that poverty can lead people to lack empathy, and be unconcerned with the suffering of others.

15. Train Deaths in Mumbai (Bombay) For example, every year in Mumbai, there are six thousand deaths caused by trains. That’s 16 deaths every day in just one city. People die by getting pushed off the platform in front of moving trains, by getting pushed off moving trains, and by getting trampled to death by people crowding onto trains. How can people be more concerned with pushing onto the train than with the people they’re pushing to death?

16. Empathy The suggested explanation is that for so long, the people in Mumbai have been so poor, that they have lost their empathetic connection to others. They don’t feel any sadness or loss when someone else is trampled or pushed off a train. Who cares about that guy? I got what I wanted!

17. A Bad Explanation? Maybe that’s true, but it might not explain the correlation between street crime and car accidents. The reason is that rich people lack empathy too. If you have lots of money, you tend to think of other people as “undeserving peasants” and not care what happens to them.

18. Fatalism Fatalism is the idea that whatever will happen will happen (“que sera sera”). The future is fixed, and there’s nothing that can change it. If poverty causes fatalism, then the poor might think that it’s not worth driving safely or using their turn signals, or whatever, because those things can’t change what will happen.

19. Fatalism This is a better explanation, because rich people tend not to be fatalists. They believe that their hard work and effort is what made them rich, not merely the unchanging hand of fate. The poor are more likely to view their situation as not due to their own laziness or incompetence, but rather due to outside forces beyond their control.

20. Multiple Factors Finally, there’s no reason to believe that the correlation might not result from all of these factors. It might be that sometimes street crime causes car crashes, sometimes car crashes cause street crime, and sometimes poverty causes both car crashes and street crime. When you “add up” all these effects, the correlation between street crime and poverty becomes strong.

21. The Point What is the point? People very often argue from a correlation claim to a causation claim. This is typically an inference to the best explanation. What best explains the correlation between chocolate eating and Nobel prizes? Maybe chocolate causes you to win Nobel prizes.

22. The Point As a critical thinker, your job is to evaluate their argument and decide whether to believe the conclusion. To do this, you have to look to see whether there are better explanations than the one being offered. “It’s more likely,” you might say, “that wealth is the common cause here.”

23. Meta-analyses

24. Meta-Analysis A meta-analysis is an analysis of analyses. In clearer terms, it is a study that looks at lots of different experiments that have been conducted on the same problem, and tries to “put together” all of the findings.

25. Motivation & Example Sometimes babies are born prematurely (like me). Unfortunately, premature babies are more likely to suffer and die.

26. Steroids In New Zealand, doctors had the idea that giving steroids to premature babies might improve their chances of survival. They did seven separate studies over nine years. Two studies showed some benefit, but five of the studies were unable to reject the null hypothesis, that steroids did not help. As a result, doctors stopped using the treatment.

27. Blobbogram

28. Blobbogram A blobbogram is a summary of a bunch of studies. Each study is represented by a line, so you can see the seven studies that were conducted on steroids in the previous slide. The line down the middle is the “no effect” line: if a study line crosses it, then that study can’t rule out the null hypothesis, that steroids are no better than placebo treatment.

29. Blobbogram Lines to the left represent positive findings. Since two studies showed positive effects, we can see two lines that are completely to the left of the “no effect” line. The length of a study’s line represents its confidence: longer lines are more uncertain. Even though 5 studies touch the “no effect” line, there seems to be a trend here: the lines tend to be to the left, positive side.

30. Meta-Analysis A meta-analysis is a way of “summing up” all the information contained in different studies of the same thing. The blue diamond to the left of the “no effect” line represents the combined meaning of all the studies: steroids work to save the lives of premature babies.

31. Large Effect Size In fact, the effect size is a reduction in the risk of death between 30% and 50%. Here’s what this means in human terms. In the US, 4 million children are born each year. 12% or about half a million of them are premature. There are about 5,000 neonatal deaths (in the first month of life) due to premature birth and complications associated with it.

32. Probabilities So the likelihood that a premature baby (in the US) will die is around 5,000 in 500,000 or 1 in 100 or 1%. This of course is right now, after we learned to use steroids. So that 1% figure is a 30% to 50% reduction of the earlier figure, which should be 1.4% to 2%.

33. The Human Cost That’s up to twice as many deaths. Between 2,150 and 5,000 babies per year died– in the US alone– because they weren’t given life-saving treatment. This happened for eight years after we had all the information to know that the treatment worked. No one had put that information together in a meta-analysis.

34. Meta-Analysis Now I hope it’s clear why these things are important. Ben Goldacre called the idea of meta-analyses an idea “that has saved the lives of more people than you will ever meet.” And it’s true! But what exactly is a meta-analysis and how does it work?

35. Literature Review The literature review is usually a search of databases containing abstracts of all the published literature or registered trials, like PubMed, Embase, or Web of Science. Researchers look for studies that have been conducted on certain topics like “breast cancer” and “wine”. This is the stage at which a meta-analysis can go really wrong for two reasons: cherry picking and the file drawer problem

36. Cherry Picking If you see cherries in the store, you might notice that most or all of them were ripe and healthy. But you couldn’t conclude that most or all cherries are ripe and healthy. These ones have been selectively picked for good condition.

37. Cherry Picking Cherry picking is another name for “the fallacy of incomplete evidence,” and it’s related to confirmation bias and selection bias. If I want to prove that a treatment works and I pick only those studies that are positive and ignore lots and lots of negative studies, then I’m “cherry picking” my studies.

38. “Systematic Review” In the “bad old days” (before the 1980s) review articles were unsystematic, meaning that people writing the reviews included some studies that were relevant, but not all. This resulted in cherry picking: later it was shown that systematic meta-analyses often had the opposite conclusions of unsystematic reviews. A systematic review looks at all the relevant studies, not just some.

39. The File Drawer Problem A second and more difficult problem for literature reviews is that the published literature is a biased sample of all the studies that have been done. As we saw last time, there is publication bias against negative results: people tend to publish positive findings, but to leave negative findings unpublished, sitting in the “file drawers” of their offices (metaphorically).

40. Detecting Publication Bias One clever way to detect whether publication bias is operative is to make a funnel plot. A funnel plot is a graph where the x-axis is the effect we’re looking for (number of premature deaths, for example) and the y-axis is how good the study is: how many people are in it, what the “variance” is, that sort of thing.

41. Good Funnel

42. Bad Funnel

43. Inverted V We always expect the best studies (at the top) to be closest to the actual effect, and the lower-quality studies to be further away from the actual effect– randomly to the left or right of it. If there’s no publication bias, we get an “inverted V” shape: lower quality studies move further away from the actual effect in both directions.

44. Publication Bias If there is publication bias, however, we get only half of an inverted V. The lower quality studies that had negative outcomes weren’t published, so there is no “left half” of the V, only a “right half. Not a lot can be done when there is serious publication bias. This is something we have to do before hand.

45. Trial Registration One common idea is that we require every trial that is conducted to be registered before it is conducted. In the registration, it describes the methods, what the researchers are trying to find, and how they intend to measure those effects. If it isn’t registered, it can’t be published when it is completed, and if it’s registered but not published, we can always ask the researchers for their data afterward.

46. Quality There is a balance that we want to maintain, however. While it’s good to include more studies in your meta-analysis, it’s bad to include low quality studies, especially since the effects of publication bias are stronger with low quality studies. In addition, low quality studies are subject to other biases as well (that’s why they’re “low quality”) so including them biases our results.

47. Selection Criteria The way to maintain this balance is to state in advance the “selection criteria”– the features an experiment must have to be included in the analysis. For example, we might only want studies with control groups, where the control is a placebo and not “no treatment”, and the participants have randomly been assigned to the groups.

48. Cherry Picking? Sometimes people with vested interests will argue that selection criteria amount to cherry picking. Studies of bad medicine often don’t have adequate randomization procedures or aren’t placebo controlled (or the placebo is not comparable to the “real” treatment– as when sugar pills are compared to acupuncture).

49. GMOs For instance, one meta-analysis showed that GMOs (genetically modified organisms) had the same nutritional value as organic produce. People who are against GMOs argued that the analysis was cherry picked because it only included RCTs and not observational studies. That’s the point! Observational studies are full of biases that RCTs don’t have.

50. Measuring Quality Even among the studies that get included, there will be differences in quality. We can measure the quality of the experiments, and report several figures: What does the meta-analysis say if only the best studies are included? What does it say if all the studies that meet the criteria are included?