Statistical Inference in Wildlife Science

1. Statistical Inference in Wildlife Science • Goals • Concerns with Nulls • Better Approaches? • Information-theoretic • Metareplication • Data dredging • Important References

2. Goal of Wildlife Research • Gain “reliable knowledge” (Romesburg 1981) • Hypothetico-deductive approach is preferred Research Hypothesis } Predictions Observed Facts Reliable Knowledge Experiment Induction Retroduction Reject Test of Statistical (Ho) Hypothesis Fail to Reject Modify Research Hypothesis Dogmatic Laws

3. Research vs. Statistical Hypotheses • H-D method includes research and statistical hypotheses • Research Hypothesis • Conjecture about a process (how nature works) based on theory (retroduction) • Statistical Hypothesis • Conjecture about a class of facts associated with the process (induction); local questions about a single population or system

4. Statisticians Have Long Debated Hypothesis Testing • Relative use of research vs. statistical hypotheses brought a long-standing debate in the world of statistics to wildlife science • We are overly concerned with testing statistical hypotheses and not concerned enough with rigorous development of, and sorting among, research hypotheses (Anderson et al. 2000)

5. Why are Statisticians Concerned with Our Over-reliance on Statistical Hypothesis Testing? • Null hypotheses are viewed incorrectly • Trivial to say there is no difference • Focus is on rejecting Ho rather than investigating the size and precision of a treatment effect • Alpha is arbitrary • Often only the P-value is reported (naked P-value) • P-value is not based on data collected, but on that not collected (probability of an observation at least as extreme as observed, given Ho) • P-value depends on N, hence rejection is certain given enough data • P-value does NOT indicate strength of Ha, but rather degree of consistency (or inconsistency) with Ho (Cherry 1998; Johnson 1999, 2002; Anderson et al. 2000; Guthery et al. 2001)

6. A Better Approach? • Focus on estimating effect size and providing a measure of its precision • Confidence Intervals do this • Rely on SE not SD, which measures variation observed in sample, not precision of estimate

7. Focus on Getting a Good Set of Biologically Reasonable Hypotheses • Embrace the concept of multiple working research hypotheses (Chamberlin 1890) rather than the single Ha vs. single Ho • Can protect research from personal bias as researchers no longer have a single favorite hypothesis they work to confirm (Guthery et al. 2001) • Formulate each hypothesis as a mathematical model • Requires close collaboration with statistician to make sure full complexity of biological hypotheses can be represented (non-linearity, etc) • Sort among multiple hypotheses using information-theoretic approach (Akaike 1973, 1974; Anderson et al. 2000; Burnham and Anderson 1998; Anderson and Burnham 2002; Anderson et al. 2001)

8. Sorting Among Research Hypotheses • Akaike’s Information Criterion (AIC) AIC Bias Best Model Amount Unexplained Variance Number of Parameters (k) (goodness of fit) + (number of parameters)

9. Rank Models • Calculate AIC and rank hypotheses (models) from best (min AIC) to worse • Single AIC not a useful value, it is relative value that is important • Akaike weights (wi) quantify the weight of evidence in favor of a model (evidence that model is best in defined set; sum of wi = 1) • Rules of Thumb • Wi >0.9 indicates a single, superior model • Relative importance of model can be indicated by change in AIC (AICi – AICmin). If Change in AIC for a model is <10, it should be considered supported by the data. • Model averaging is a powerful way to estimate parameters and their precision • Average parameter value is weighted average (using wi) of parameters (Øaverage = sum wiØi )

10. Some Issues with I-T Approach (Guthery et al. 2001; Robinson and Wainer 2002) • Method is parametric • Requires assumptions about distributions to be met • Definition of research hypotheses defines conclusions • One of those in the group of alternative models will have the minimum AIC • Need to make sure no trivial hypotheses are in set • Hypotheses should reflect plausible, but different, ways that nature works (i.e., be true research hypotheses, not statistical hypotheses) • Null effects are not necessarily trivial, must be modeled if there is good reason • Frequentist statistics are appropriate for analysis of well designed experiments

11. Better Use of P-value(Fisher 1925; Robinson and Wainer 2002) • IF you use frequentist approach, then: • Follow Fisher’s lead and use p-values to screen for potentially real or useful associations that have merit for future investigation, rather than using them to identify end points (significant findings to draw conclusions from) of an investigation

12. Better Use of P-value(Fisher 1925; Robinson and Wainer 2002) • IF you use frequentist approach, then: • Report actual p-value and effect size plus measure of precision • Do not make reject / fail-to-reject decisions • If P<0.05, report evidence of effect and look to confirm with other studies • If 0.2>P>0.05, report evidence exists for further testing of hypothesis with improved design (replication). State result “leans” in a certain direction. • If P>0.2, report that if there is an effect, it is too small to detect with the current experimental design • If you are doing a 1-time experiment, then α should be reduced well below 0.05 • Do not interpret P as the probability of Ho given the data, it is the probability of the data, given a true Ho • If you want to discuss likelihood of a hypothesis, then I-T or Bayesian approaches are more appropriate

13. Metareplication (Johnson 2002) • This approach gets away from individual P-values by focusing on making inference in the context of prior related findings • A Bayesian approach following Fisher’s lead • Search for multiple studies to point in a common direction rather than a single definitive study with low p-value to show the direction • Replication of studies (metareplication) is the key • Exploit value of small studies, each of which may not be able to make a definitive conclusion • Truth lies at the “intersection of independent lies” (Levins 1966) • Although independent studies each may suffer from various shortcomings (small n, etc.), if they paint substantially similar pictures, we have confidence in what we see

14. Making Management Recommendations • Place less emphasis on the significant finding of an individual study • Use estimates of effect size and precision from individual studies in meta-analysis to determine consistent effects before making management recommendations • Look for truly replicated studies with consistent findings • Different methods, different locations, different observers

15. Dredging Data Along the Way • Dredging data is not bad, it is the creative process, but analyzing dredged data with traditional statistical methods is a violation of assumptions • Surprising findings should be heralded and used to stimulate new hypotheses and experiments • Put dredged findings in Discussion not Results • Admit it when you dredge • Use dredging to screen for possible effects to be considered in future studies

16. “Any single study can yield a p-value, but only consistency among replicated studies will advance our science” (Johnson 2002)

17. If YOU Are Doing Research, YOU MUST Read: • Anderson, DR, Link, WA, Johnson, DH, and KP Burnham. 2001. Suggestions for presenting the results of data analysis. J. Wildlife Manage. 65:373-378. • Anderson, DR, Burnham, KP, and WL Thompson. 2000. Null hypothesis testing: problems, prevalence, and an alternative. J. Wildlife Manage. 64:912-923. • Burnham, KP and DR Anderson. 1998. Model selection and inference: a practical information-theoretic approach. Springer-Verlag, New York. • Chamberlin, TC. 1890. The method of multiple working hypotheses. Science 148:754-759 (reprinted there) • Cherry, S. 1998. Statistical tests in publications of The Wildlife Society. Wildlife Society Bulletin 26:947-953. • Johnson, DH. 1999. The insignificance of statistical significance testing. J. Wildlife Management 63:763-772. • Robinson, DH and H. Wainer. 2002. On the past and future of null hypothesis significance testing. J. Wildlife Management 66:263-271. • Fisher, R.A. 1925. Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society 22:700-725. • Fisher, RA. 1928. Statistical methods for research workers. 2nd edition. Oliver and Boyd. London. • Anderson, DR and KP Burnham. 2002. Avoiding pitfalls when using information-theoretic methods. J. Wildlife Management 66:912-918. • Akaike, H. 1973. Information theory as an extension of the maximum likelihood principle. Pp 267-281. in. BN Petrov and F Csaki, eds. Second international symposium on information theory. Akademiai Kiado, Budapest. • Akaike, H. 1974. A new look at the statistical model identification. IEEE Transactions on automatic control AC 19:716-723. • Johnson, DG. 2002. The importance of replication in wildlife research. J. Wildlife Management 66:919-932. • Johnson, DG. 2002. The role of hypothesis testing in wildlife science. J. Wildlife Management 66:272-276. • Guthery, FS, JJ Lusk, and MJ Peterson. 2001. The fall of the null hypothesis: liabilities and opportunities. J. Wildlife Management 65:379-384. • Hurlbert, SH. 1984. Pseudoreplication and the design of ecological field experiments. Ecological Monographs 54:187-211. • Romesburg, HC. 1981. Wildlife science: gaining reliable knowledge. J. Wildlife Management. 45:293-313.