1 / 24

Update on the Goodness of Fit Toolkit

Update on the Goodness of Fit Toolkit. B. Mascialino, A. Pfeiffer, M.G. Pia , A. Ribon, P. Viarengo. PHYSTAT 2005 Oxford, 11-15 September 2005. http://www.ge.infn.it/geant4/analysis/HEPstatistics http://www.ge.infn.it/statisticaltoolkit.

jessecastro
Download Presentation

Update on the Goodness of Fit Toolkit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Update on theGoodness of Fit Toolkit B. Mascialino, A. Pfeiffer, M.G. Pia, A. Ribon, P. Viarengo PHYSTAT 2005 Oxford, 11-15 September 2005 http://www.ge.infn.it/geant4/analysis/HEPstatistics http://www.ge.infn.it/statisticaltoolkit

  2. Electromagnetic models in Geant4 w.r.t. NIST reference ESA Bepi Colombo mission to Mercury test beam at Bessy NIST data Geant4 Standard Geant4 LowE Photon attenuation coefficient, Al Fluorescence spectrum from Icelandic basalt (Mars-like rock): experimental data and simulation Historical background… Validation of Geant4 physics models through comparison of simulation vs experimental data or reference databases

  3. Some use cases The test statistics computation concerns the agreement between the two samples’ empirical distribution functions • Regression testing • Throughout the software life-cycle • Online DAQ • Monitoring detector behaviour w.r.t. a reference • Simulation validation • Comparison with experimental data • Reconstruction • Comparison of reconstructed vs. expected distributions • Physics analysis • Comparisons of experimental distributions (ATLAS vs. CMS Higgs?) • Comparison with theoretical distributions (data vs. Standard Model)

  4. G.A.P Cirrone, S. Donadio, S. Guatelli, A. Mantero, B. Mascialino, S. Parlati, M.G. Pia, A. Pfeiffer, A. Ribon, P. Viarengo “A Goodness-of-Fit Statistical Toolkit” IEEE- Transactions on Nuclear Science (2004), 51 (5): 2056-2063. More information on the project web site

  5. Vision of the project • Basicvision • General purpose tool • Toolkit approach (choice open to users) • Open source product • Independent from specific analysis tools • Easily usable in analysis and other tools • Rigorous software process Clearly define scope, objectives Software quality Flexible, extensible, maintainable system Build on a solid architecture

  6. Software process guidelines • Adopt a process • the key to software quality... • Unified Process, specifically tailored to the project • practical guidance and tools from the RUP • both rigorous and lightweight • mapping onto ISO 15504 (and CMM) • Incremental and iterative life-cycle • 1st cycle: 2-sample GoF tests • 1-sample GoF in preparation

  7. Architectural guidelines • The project adopts a solid architectural approach • to offer the functionalityand the quality needed by the users • to be maintainable over a large time scale • to be extensible, to accommodate future evolutions of the requirements • Component-based architecture • to facilitate re-use and integration in diverse frameworks • layer architecture pattern • core component for statistical computation • independent components for interface to user analysis environments • Dependencies • no dependence on any specific analysis tool • can be used by any analysis tools, or together with any analysis tools • offer various specific user layers for usage with AIDA, ROOT, etc.

  8. User Layer • Simple user layer • Shields the user from the complexity of the underlying algorithms and design • Only deal withthe user’sanalysis objectsand choice ofcomparison algorithm

  9. GoF algorithms (currently implemented) • Algorithms for binned distributions • Anderson-Darling test • Chi-squared test • Fisz-Cramer-von Mises test • Tiku test(Cramer-von Mises test in chi-squared approximation) • Algorithms for unbinned distributions • Anderson-Darling test • Cramer-von Mises test • Goodman test(Kolmogorov-Smirnov test in chi-squared approximation) • Kolmogorov-Smirnov test • Kuiper test • Tiku test(Cramer-von Mises test in chi-squared approximation)

  10. Recent extensions: algorithms • Fisz-Cramer-von Mises test and Anderson-Darling test • exact asymptotic distribution (earlier: critical values) • Tiku test • Cramer-von Mises test in a chi-squared approximation • New tests: weighted Kolmogorov-Smirnov, weighted Cramer-von Mises • various weighting functions available in literature • In preparation • Watson test (can be applied in case of cyclic observations, like Kuiper test) • Girone test • It is the most complete software for the comparison of two distributions, even among commercial/professional statistics tools • goal: provide all 2-sample GoF algorithms existing in statistics literature • Publication in preparation to describe the new algorithms

  11. Recent extensions: user layer • First release: user layer for AIDA analysis objects • LCG Architecture Blueprint, Geant4 requirement • July 2005: added user layer for ROOT histograms • in response to user requirements • Other user layer implementations foreseen • easy to add • sound architecture decouples the mathematical component and the user’s representation of analysis objects • different requirements from various user communities: satisfy them without introducing dependencies on any analysis tools

  12. Software release • Releases are publicly downloadable from the web • code, documentation etc. • For the convenience of LCG users, releases are also distributed with LCG AA software as “external contributions” • Also ported to Java, distributed with JAS • Release with new algorithms planned in autumn • + publication on recent extensions • Releases include extensive user documentation • statistics algorithms • how to use the software • The project is systematically accompanied by publications on refereed journals to document the recognition of its scientific value

  13. Examples of Usage • Geant4 physics validation • rigorous approach: quantitative evaluation of Geant4 physics models with respect to established reference data • see for instance: K. Amako et al., Comparison of Geant4 electromagnetic physics models against the NIST reference dataIEEE Trans. Nucl. Sci. 52- 4 (2005) 910-918 • LCG Simulation Validation project • see for instance: A. Ribon, Testing Geant4 with a simplified calorimeter setup, http://www.ge.infn.it/geant4/events/july2005 • CMS • validation of “new” histograms w.r.t. “reference” ones in OSCAR Validation Suite • Usage also in space science, medicine etc.

  14. Power of GoF tests • Do we really need such a wide collection of GoF tests? Why? • Which is the most appropriate test to compare two distributions? • How “good” is a test at recognizing real equivalent distributions and rejecting fake ones? Which test to use?

  15. Systematic study of GoF tests • No comprehensive study of the relative power of GoF tests exists in literature • novel research in statistics (not only in physics data analysis!) • Systematic study of allexisting GoF tests in progress • made possible by the extensive collection of tests in the Statistical Toolkit • Provide guidance to the users based on sound quantitative arguments • Preliminary results available • Publication in preparation

  16. Parent distribution 1 Parent distribution 2 GoF test Sample 1 n Sample 2 m Confidence Level = 0.05 # pseudoexperiments with p-value < (1-CL) Power = # pseudoexperiments Method for the evaluation of power Pseudoexperiment: a random drawing of two samples from two parent distributions N=1000 Monte Carlo replicas For each test, the p-value computed by the GoF Toolkit derives from the analytical calculation of the asymptotic distribution, often depending on the samples sizes

  17. Gaussian Uniform Exponential Double exponential Cauchy Contaminated Normal Distribution 1 Contaminated Normal Distribution 2 Parent distributions Also Breit-Wigner, other distributions being considered

  18. Characterization of distributions Skewness Tailweight

  19. Comparative evaluation of tests Preliminary Tailweight Skewness

  20. Preliminary results • No clear winner for all the considered distributions in general • the performance of a test depends on its intrinsic features as well as on the features of the distributions to be compared • Practical recommendations • first classify the type of the distributions in terms of skewness and tailweight • choose the most appropriate test given the type of distributions • Systematic study of the power in progress • for both binned and unbinned distributions • Topic still subject to research activity in the domain of statistics • Publication in preparation

  21. Outlook • 1-sample GoF tests (comparison w.r.t. a function) • Comparison of two/multi-dimensional distributions • Systematic study of the power of GoF tests • Goal to provide an extensive set of algorithms so far published in statistics literature, with a critical evaluation of their relative strengths and applicability • Treatment of errors, filtering • New release coming soon • New papers in preparation • Other components beyond GoF? Suggestions are welcome…

  22. Conclusions • A novel, complete software toolkit for statistical analysis is being developed • rich set of algorithms • sound architectural design • rigorous software process • A systematic study of the power of GoF tests is in progress • unexplored area of research • Application in various domains • Geant4, HEP, space science, medicine… • Feedback and suggestions are very much appreciated • The project is open to developers interested in statistical methods

More Related