Steps 3 & 4: Evaluating types of evidence for the Truckee River case study

1. Steps 3 & 4: Evaluating types of evidence for the Truckee River case study

2. Detect or Suspect Biological Impairment Stressor Identification Define the Case List Candidate Causes Decision-maker and Stakeholder Involvement As Necessary: Acquire Data and Iterate Process Step 3: Evaluate Data from the Case Step 4: Evaluate Data from Elsewhere Evaluate Data from Elsewhere Identify Probable Cause Identify and Apportion Sources Management Action: Eliminate or Control Sources, Monitor Results Biological Condition Restored or Protected

3. Use all available types of evidence to make an inferential assessment • Types of evidence using data from the case • Spatial/temporal co-occurrence • Evidence of exposure or biological mechanism • Causal pathway • Stressor-response relationships from the field • Manipulation of exposure • Laboratory tests of site media • Temporal sequence • Verified predictions • Symptoms • Types of evidence using data from elsewhere • Stressor-response relationships from other field studies • Stressor-response relationships from laboratory studies • Stressor-response relationships from ecological simulation models • Mechanistically plausible cause • Manipulation of exposure at other sites • Analogous stressors italics indicates commonly available types of evidence

4. Basic analysis strategy • Develop as many types of evidence, for as many candidate causes, as you can • you won’t have all types of evidence, for all candidate causes • most effective when you can compare results across candidate causes • Work through one type of evidence, then set it aside • avoid cognitive overload • Show your work • make your process transparent & reproducible • make use of appendices

5. Let’s begin by figuring out what types of evidence we have for the Truckee…

6. General vs. specific causation • General – Does C cause E? • Does smoking cause lung cancer? • Does increased water temperature reduce bull trout abundance in rivers? • Specific – Did C cause E? • Did smoking cause lung cancer in Ronald Fisher? • Did increased water temperature reduce bull trout abundance in my stream?

7. Specific causation: using data from the case

8. SUPPORTS Want paired measurements of proximate stressors & biological impairments, at locations where impairments are & are not observed. WEAKENS Spatial/temporal co-occurrence

9. Want paired measurements of other steps in causal pathway & biological impairments, at locations where impairments are & are not observed. Causal pathway

10. Want paired measurements of proximate stressors (or other steps in causal pathway) & biological impairments, at varying levels of exposure. Stressor-response relationships from the field

11. Other types of evidence using data from case

12. What types of evidence do we have, using data from the case?

13. General causation: using data from elsewhere?

14. Stressor-response relationships from other field studies

15. Stressor-response relationships from the lab

16. Other types of evidence using data from elsewhere

17. What types of evidence do we have, using data from elsewhere?

18. Now that we know what data we have, how do we analyze it?

19. Spatial co-occurrence Do your impairment and your stressor co-occur in space? To Do: • Load relevant data file • Merge files • Make boxplots for each candidate cause Select ‘reference’ and impaired sites • Fill in worksheet

20. Causal Pathway Does your data support the steps in the causal path between the stressor and the impairment? To Do: • Return to the conceptual diagram • Identify the steps in the causal pathway • Construct table to show whether data supports the steps between the stressor and the impairment • Fill in worksheet

21. Verified Prediction - Traits Do data support predictions based on stressor’s mode of action? To Do: • Load relevant data file • Merge files • Make boxplot Select ‘reference’ and impaired sites • Fill in worksheet

22. Verified Prediction - PECBO Do data support predictions based on stressor’s mode of action? To Do: • Load relevant data file • Merge files • Run PECBO • Load PECBO results file into CADStat • Merge files • Make boxplot • Sed • STRMTEMP • Fill in worksheet

23. Stressor-response from elsewhere Does impairment decrease as exposure to the stressor decreases (or increases as exposure increases)? To Do: • Listen and ask lots of questions • Fill in the worksheet

24. Randomized, controlled experiments The scientific standard for establishing cause and effect Key elements: • Replication: use of multiple test units (e.g. tanks, sites) • Controls: differ only by absence of the treatment • Randomization: random assignment of test units to “control” or “treated” status • Statisticalanalysis: estimate treatment effect (causal)

25. Observational studies Often the only option for large-scale field studies Key elements: • Replication: collect data from multiple test units • Controls: ? • Randomization: ? • Statisticalanalysis: identify associations among variables of interest (non-causal) None None

26. control realism, scale Trade offs: control vs. realism, scale Lab Experiment Field Experiment Observational Study

27. Biomonitoring = Observational Issues for causal analysis: • Estimates of stressor effects are confounded by covarying factors • Analyst can’t randomly assign treatments (stressors) to sites * Reference sites are not experimental controls

28. Analogous to clinical trials Does smoking cause lung cancer? • Estimates of stressor effects are confounded by covarying factors • Analyst can’t randomly assign treatments (stressors) to subjects * Non-smokers without lung cancer are not experimental controls

29. Example using western EMAP* Using propensity scores to infer cause-effect relationships in observational data • Analysis and slides by Lester Yuan (USEPA), Amina Pollard (USEPA), and Daren Carlisle (USGS) • Original presentation given at North American Benthological Society conference, May 2008 *EPA Environmental Monitoring and Assessment Program (EMAP)

30. EMAP-West Study Area • Measurements Collected: • Macroinvertebrates • Substrate composition • (SED) • Stream temperature • (STRMTEMP) • N = 838 Data collected by the EPA Environmental Monitoring and Assessment Program (EMAP)

31. Total N vs. total taxon richness SLOPE = -16.5 Data from EMAP Western Pilot

32. Total N covaries with many other factors

33. Include covariates in the regression model to control for their effect. SLOPE = -16.5 Multiple linear regression Correlation of Total Richness and Total N (ug/L) Regression model includes: %agriculture, %urban, grazing intensity, %sands/fines, stream temperature, and log conductivity. SLOPE = -9.4

34. Must assume that linear relationships are appropriate for all covariates. Regression model may extrapolate. Inclusion of certain variables may “mask” true effect: e.g., part of the effect of agriculture may be attributed to total N Potential issues with multiple regression

35. Alternate approach: Stratify dataset r = 0.64 r = -0.01 r = 0.15 r = 0.27

36. Model richness vs. total N within strata SLOPE = -10.7 SLOPE = -12.3 SLOPE = -9.7 How do we simultaneously stratify on many different covariates?

37. Method developed in epidemiology to retroactively control for confounding effects in observational studies Sometimes called a quasi-experiment Intuitively: Model the magnitude of treatment (e.g. nutrient concentration) as a function of the covariates. The predicted magnitude of treatment at each site is its propensity score. Stratify the total set of observations by the propensity scores (i.e., group sites with similar scores). Six strata are typically used. Within each stratum, sites having different treatment levels (e.g. high vs. low nutrients) may be considered to have been “randomly assigned” to those treatment levels, because covariates have effectively been controlled by propensity score matching of “treated” and “control” sites. Propensity Score Matching

38. Propensity Score Model Total N = f(percent agriculture, percent urban, grazing intensity, percent sand/fines, stream temperature, log conductivity,elevation, log catchment area, canopy cover, sampling day)

39. Define 6 strata based on propensity scores

40. Covariate values within strata Percent agriculture Grazing intensity

41. Stratification by propensity score controls covariance of all modeled variables After stratification

42. Total N vs. total taxon richness SLOPE = -16.5 Data from EMAP Western Pilot

43. Total N vs. total richness: Stratified SLOPE = -3.3 (n.s.) SLOPE = -7.1* SLOPE = -7.1* SLOPE = -10.0*** SLOPE = -10.5*** SLOPE = -8.1***