1 / 81

Nonparametric Methods III

Nonparametric Methods III. Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University hslu@stat.nctu.edu.tw http://tigpbp.iis.sinica.edu.tw/courses.htm. PART 4: Bootstrap and Permutation Tests. Introduction References Bootstrap Tests Permutation Tests Cross-validation

garin
Download Presentation

Nonparametric Methods III

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Nonparametric Methods III Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University hslu@stat.nctu.edu.tw http://tigpbp.iis.sinica.edu.tw/courses.htm

  2. PART 4: Bootstrap and Permutation Tests • Introduction • References • Bootstrap Tests • Permutation Tests • Cross-validation • Bootstrap Regression • ANOVA

  3. References • Efron, B.; Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC. • http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-bootstrapping.pdf • http://cran.r-project.org/bin/macosx/2.1/check/bootstrap-check.ex • http://bcs.whfreeman.com/ips5e/content/cat_080/pdf/moore14.pdf

  4. Hypothesis Testing (1) • A statistical hypothesis test is a method of making statistical decisions from and about experimental data. • Null-hypothesis testing just answers the question of “how well the findings fit the possibility that chance factors alone might be responsible.” • This is done by asking and answering a hypothetical question. http://en.wikipedia.org/wiki/Statistical_hypothesis_testing

  5. Hypothesis Testing (2) • Hypothesis testing is largely the product of Ronald Fisher, Jerzy Neyman, Karl Pearson and (son) Egon Pearson. Fisher was an agricultural statistician who emphasized rigorous experimental design and methods to extract a result from few samples assuming Gaussian distributions. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an (extended) hybrid of the Fisher vs. Neyman/Pearson formulation, methods and terminology developed in the early 20th century.

  6. Hypothesis Testing (3)

  7. Hypothesis Testing (4)

  8. Hypothesis Testing (5)

  9. Hypothesis Testing (7) • Parametric Tests: • Nonparametric Tests: • Bootstrap Tests • Permutation Tests

  10. Confidence Intervals vs. Hypothesis Testing (1) • Interval estimation ("Confidence Intervals") and point estimation ("Hypothesis Testing") are two different ways of expressing the same information. http://www.une.edu.au/WebStat/unit_materials/ c5_inferential_statistics/confidence_interv_hypo.html

  11. Confidence Intervals vs. Hypothesis Testing (2) • If the exact p-value is reported, then the relationship between confidence intervals and hypothesis testing is very close. However, the objective of the two methods is different: • Hypothesis testing relates to a single conclusion of statistical significance vs. no statistical significance. • Confidence intervals provide a range of plausible values for your population. http://www.nedarc.org/nedarc/analyzingData/ advancedStatistics/convidenceVsHypothesis.html

  12. Confidence Intervals vs. Hypothesis Testing (3) • Which one? • Use hypothesis testing when you want to do a strict comparison with a pre-specified hypothesis and significance level. • Use confidence intervals to describe the magnitude of an effect (e.g., mean difference, odds ratio, etc.) or when you want to describe a single sample. http://www.nedarc.org/nedarc/analyzingData/ advancedStatistics/convidenceVsHypothesis.html

  13. P-value http://bcs.whfreeman.com/ips5e/content/cat_080/pdf/moore14.pdf

  14. Achieved Significance Level (ASL) https://www.cs.tcd.ie/Rozenn.Dahyot/453Bootstrap/05_Permutation.pdf

  15. Methodology Flowchart R code Bootstrap Tests

  16. Bootstrap Tests • Beran (1988) showed that bootstrap inference is refined when the quantity bootstrapped is asymptotically pivotal. • It is often used as a robust alternative to inference based on parametric assumptions. http://socserv.mcmaster.ca/jfox/Books/Companion/appendix-bootstrapping.pdf

  17. Hypothesis Testing by a Pivot http://en.wikipedia.org/wiki/Pivotal_quantity

  18. One Sample Bootstrap Tests • T statistics can be regarded as a pivot or an asymptotic pivotal when the data are normally distributed. • Bootstrap T tests can be applied when the data are not normally distributed.

  19. Bootstrap T tests • Flowchart • R code

  20. Flowchart of Bootstrap T Tests Bootstrap B times

  21. Bootstrap T Tests by R

  22. An Example of Bootstrap T Tests by R

  23. Bootstrap Tests by The “BCa” • The BCa percentile method is an efficient method to generate bootstrap confidence intervals. • There is a correspondence between confidence intervals and hypothesis testing. • So, we can use the BCa percentile method to test whether H0 is true. • Example: use BCa to calculate p-value

  24. BCa Confidence Intervals: • Use R package “boot.ci(boot)” • Use R package “bcanon(bootstrap)” • http://qualopt.eivd.ch/stats/?page=bootstrap • http://www.stata.com/capabilities/boot.html

  25. http://finzi.psych.upenn.edu/R/library/boot/DESCRIPTION

  26. An Example of “boot.ci(boot)” in R

  27. http://finzi.psych.upenn.edu/R/library/bootstrap/DESCRIPTION

  28. An example of “bcanon(bootstrap)” in R

  29. BCa byhttp://qualopt.eivd.ch/stats/?page=bootstrap

  30. Use BCa to calculate p-value by R

  31. Two Sample Bootstrap Tests • Flowchart • R code

  32. Flowchart of Two-Sample Bootstrap Tests combine m+n=N Bootstrap B times

  33. Two-Sample Bootstrap Tests by R

  34. Output (1)

  35. Output (2)

  36. Permutation Tests • Methodology • Flowchart • R code

  37. Permutation • In several fields of mathematics, the term permutation is used with different but closely related meanings. They all relate to the notion of (re-)arranging elements from a given finite set into a sequence. http://en.wikipedia.org/wiki/Permutation

  38. Permutation Tests • Permutation testisalso called a randomization test, re-randomization test, or an exact test. • If the labels are exchangeable under the null hypothesis, then the resulting tests yield exact significance levels. • Confidence intervals can then be derived from the tests. • The theory has evolved from the works of R.A. Fisher and E.J.G. Pitman in the 1930s. http://en.wikipedia.org/wiki/Pitman_permutation_test

  39. Applications of Permutation Tests (1) We can use a permutation test only when we can see how to resample in a way that is consistent with the study design and with the null hypothesis. http://bcs.whfreeman.com/ips5e/content/ cat_080/pdf/moore14.pdf

  40. Applications of Permutation Tests (2) • Two-sample problemswhen the null hypothesis says that the two populations are identical. We may wish to compare population means, proportions, standard deviations, or other statistics. • Matched pairs designswhen the null hypothesis says that there are only random differences within pairs. A variety of comparisons is again possible. • Relationships between two quantitative variableswhen the null hypothesis says that the variables are not related. The correlation is the most common measure of association, but not the only one. http://bcs.whfreeman.com/ips5e/content/ cat_080/pdf/moore14.pdf

  41. Inference by Permutation Tests https://www.cs.tcd.ie/Rozenn.Dahyot/453Bootstrap/05_Permutation.pdf

  42. Flowchart of The Permutation Test for Mean Shift in One Sample Partition 2 subset B times (treatment group) (treatment group) (control group) (control group)

  43. An Example for One Sample Permutation Test by R http://mason.gmu.edu/~csutton/ EandTCh15a.txt

  44. An Example of Output Results

  45. Flowchart of The Permutation Test for Mean Shift in Two Samples combine m+n=N Partition subset B times treatment subgroup control subgroup treatment subgroup control subgroup

  46. Bootstrap Tests vs. Permutation Tests • Very similar results between the permutation test and the bootstrap test. • is the exact probability when . • is not an exact probability but is guaranteed to be accurate as an estimate of the ASL, as the sample size B goes to infinity. https://www.cs.tcd.ie/Rozenn.Dahyot/453Bootstrap/05_Permutation.pdf

  47. Cross-validation • Methodology • R code

  48. Cross-validation • Cross-validation, sometimes called rotation estimation, is the statistical practice of partitioning a sample of data into subsets such that the analysis is initially performed on a single subset, while the other subset(s) are retained for subsequent use in confirming and validating the initial analysis. • The initial subset of data is called the training set. • the other subset(s) are called validation or testing sets. http://en.wikipedia.org/wiki/Cross-validation

  49. Overfitting Problems • In statistics, overfitting is fitting a statistical model that has too many parameters. • When the degrees of freedom in parameter selection exceed the information content of the data, this leads to arbitrariness in the final (fitted) model parameters which reduces or destroys the ability of the model to generalize beyond the fitting data. • The concept of overfitting is important also in machine learning. • In both statistics and machine learning, in order to avoid overfitting, it is necessary to use additional techniques (e.g. cross-validation, early stopping, Bayesian priors on parameters or model comparison), that can indicate when further training is not resulting in better generalization. • http://en.wikipedia.org/wiki/Overfitting

More Related