1 / 42

Ontology Evolution and Regression Analysis Insights into Ontology Regression Testing

Ontology Evolution and Regression Analysis Insights into Ontology Regression Testing. Maria Copeland Rafael Goncalvez Robert Stevens Bijan Parsia Uli Sattler. Motivation. Current studies of Ontology Evaluation tend to: Focus on individual ontology versions

holden
Download Presentation

Ontology Evolution and Regression Analysis Insights into Ontology Regression Testing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ontology Evolution and Regression AnalysisInsights into Ontology Regression Testing Maria Copeland Rafael Goncalvez Robert Stevens Bijan Parsia Uli Sattler

  2. Motivation Current studies of Ontology Evaluation tend to: • Focus on individual ontology versions • Focus on shifts in the gross statistics In either case we don’t get objective and systematic evaluations of the life span of the ontology

  3. Our goal is to extract insightful and useful information out of all the existing versions of an ontology

  4. Ontology Testing Challenge • How do we systematically identify test areas? • How do we systematically analyse change impacts to the ontology? How can we effectively minimise testing efforts and cost and still achieve adequate testing coverage

  5. Software Testing

  6. Software Regression TestingWhat is it? • It is a test activity to systematicallyre-test existing components after software changes • It test against current and updated requirements

  7. Software Regression TestingTesting Aspects • Testing at the functional requirements • Unit Level • System Level • Testing at the non-functional requirements

  8. Software Regression TestingTesting Plan • Defines testing criteria • Identifies test area or components • Test execution strategy • Test evaluation strategy • Updates test and other relevant documentation

  9. Software Regression TestingProcess Change Detection Impact Analysis Define Test Plan Build Test Suite Run Test Evaluate Results Report Results

  10. Ontology Regression Testing?

  11. Change Detection • Explicit Changes • Asserted logical and annotation axioms • Properties • Classes • Implicit Changes? • Subsumption changes • Entailment changes

  12. Impact Analysis • Previous Version or all versions? • Intentional Difference analysis? • Justifications analysis? • Information Content? • Asserted content? • Entailed content? • Requirements Impact? • Functional and Non-functional? • Which ones do we test?

  13. Define Test Plan • Test criteria? • Test area? Do we have test areas? • How can the test be systematically run? • Can results be interpreted?

  14. Can we systematically: • Build test suites? • Run tests? • Evaluate results? • Re-run tests if necessary?

  15. Manual vs. Automated Test Suites • Manually test cases • Check against a methodology • Eyeballing • Automated test cases • Satisfiability • Inconsistency

  16. Manual vs. Automated Test Suites • Manually test cases • Time consuming • Subjective • Unsystematic • Automated test cases • Reasoner based • Limited in scope

  17. Can we expand the range of automatic test suites?

  18. YESBy Analysing Ontology Dynamics

  19. What are Ontology Dynamics? • Periods of growth, decline, and stability • Axioms presence • Types of axioms presence (e.g. continual, interrupted) • Sequence editing types and patterns

  20. Axiom Life Span - We expect: Axioms with Constant Unchanged Presence αi O1 Oi Ontology Versions

  21. Axiom Life Span - We expect: Axioms that are Modified αi Split or Merge O1 Oi Oi+n Ontology Versions

  22. Axiom Life Span - We expect: Axioms that Enter and Leave the Ontology αi O1 Oi Oi+n Ontology Versions

  23. NCIt Ontology Dynamics

  24. National Cancer Institute Thesaurus (NCIt) • The National Cancer Institute (NCI) is a U.S. government funded organisation for the research of causes, treatment, and prevention of cancer • The NCIt is an ontology written in the Web Ontology Language (OWL) which supports the development and maintenance of a controlled vocabulary about cancer research • Multiple publications about process, quality control, usage, and critiques • Publicly available monthly releases and concept change logs Rich source of ontology evolution data

  25. NCIt Dynamics – Axioms Life Span Analysis

  26. NCIt Change Dynamics – Detailed View of Axiom Life Span Top Ten Frequency Distributions

  27. NCIt Axiom Life Span: Axioms with Constant Unchanged Presence 6.73% αi O1 O103 NCIt 2003 – 2012 103 Versions

  28. NCIt Axiom Life Span: Axioms that Enter and Leave the Ontology αi O1 Oi Oi+n NCIt 2003 – 2012 103 Versions

  29. NCIt Axiom Life Span: Axioms with Gaps between Unchanged Presence Effectually Added Effectually Removed Effectually Added Effectually Removed αi O I≠ αi αi O I≠ αi O1 Oi Oi+n Ok Ok+n NCIt 2003 – 2012 103 Versions

  30. NCIt Axiom Life Span: Axioms with Gaps between Unchanged Presence O I≠ αi Effectually Added Ineffectually Removed Effectually Removed Effectually Added αi αi O I= αi O1 Oi Oi+n Ok Ok+n NCIt 2003 – 2012 103 Versions

  31. NCIt Axiom Life Span: Axioms with Gaps between Unchanged Presence Ineffectually Removed Ineffectually Added Ineffectually Added Ineffectual Removed αi • O I= αi αi • O I= αi O1 Oi Oi+n Ok Ok+n NCIt 2003 – 2012 103 Versions

  32. NCIt Dynamics – Editing Patterns

  33. NCIt Regression Analysis – Main Finding This means that we are able to identify ‘bugs’, the sequence pattern of these bugs, and their location!

  34. NCIt Regression Analysis: Indicative of Faults In Sequence of Changes Effectually Added Effectually Added Ineffectually Removed Effectually Removed Effectually Removed Effectually Added Effectually Removed Effectually Added ? ? ✓ ✗ αi αi αi αi O I= αi O I≠ αi O I≠ αi O1 Oi Oi+n Ok Ok+n O1 Oi Oi+n Ok Ok+n NCIt 2003 – 2012 103 Versions NCIt 2003 – 2012 103 Versions ✗ ? ✓ ?

  35. NCIt Regression Analysis: Suggestive of Faults In Sequence of Changes Ineffectually Removed Effectually Added O I= αi αi O1 Ok Ok+n NCIt 2003 – 2012 103 Versions Refactoring

  36. NCIt Regression Analysis: Suggestive of Faults In Sequence of Changes Ineffectually Added Ineffectually Removed Ineffectually Removed Effectually Added O I= αi αi αi ? O1 Ok Oi Oi+n Ok+n NCIt 2003 – 2012 103 Versions Ineffectually Added O I= αi Ok O1 NCIt 2003 – 2012 103 Versions

  37. From Change Dynamics to Ontology Regression Testing

  38. Systematically Build Test Suites Indicative of Faults In Sequence of Changes

  39. Automated Test Suites - Fault Detection • It provides systematic regression test for all version of the ontology • It conclusively identifies content regression and content refactoring • It suggests other faults based on regression sequence It is efficient and cheap to run

  40. And there are still more potential benefits …

  41. Automated Test Suites - Fault Detection • Entailment Set Studies • Sub Domain Dynamics • Ontology Classes Dynamics …

  42. Thanks

More Related