250 likes | 365 Views
MADARA KATS: Automated Testing with Distributed Knowledge and Reasoning. James Edmondson Vanderbilt Institute for Software Integrated Systems Mail: james.r.edmondson@vanderbilt.edu Project site: http://madara.googlecode.com. KATS: Table of Contents. Scenario. Overview. Modeling.
E N D
MADARA KATS:Automated Testing with Distributed Knowledge and Reasoning James Edmondson Vanderbilt Institute for Software Integrated Systems Mail: james.r.edmondson@vanderbilt.edu Project site: http://madara.googlecode.com
KATS: Table of Contents Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Motivating scenario • Solution overview • Modeling tests • Disseminating and evaluating test knowledge • Automating Android smartphone tests with MAML • Demonstration • Conclusions
KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Three computers and two smart phones connected to a network • The two smart phones are connected to two of the computers network
KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • The phones communicate with system services on the server • The phones must wait for the system services to warm up before connecting (5 seconds) server System Services network
KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • A race condition exists if App 2 is started immediately after App 1 • How do we automatedly test for this type of scenario? server Phone App 1 System Services network Phone App 2
KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • If the test fails, we may have to run a separate test on another host server Phone App 1 System Services network Process Results Process Results Phone App 2 Phone App 3 Process Results Process Results • After the tests are performed, each host must process its log results
KATS: Motivating Scenario Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Scenario Requirements: • Ability to sequence tests, preferably without a need for a centralized server (single point of failure) • Ability to sequence different types of programs (e.g. Python results parsers, C++ services, and Java smart phone applications) • Ability to disseminate knowledge of test failure and success • Configurable knowledge reasoning service to make sense of the incoming information • Portable to Windows, Linux and other operating systems
MADARA: Architectural Overview Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • EXTERNAL DEPENDENCIES • C++ for speed and portability • ACE for portable threads and OS calls • DDS for transport of knowledge • Android SDK for Android smartphone dev • Monkeyrunner for sending events to phones KATS MAML KaRL Monkeyrunner ACE DDS Android SDK C++ Python Java • DEVELOPED TECHNOLOGIES • MAMLfor instrumentation of phones • KaRLfor knowledge and reasoning • KATS for automated testing
KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The barrier condition is met when all potentially distributed entities participating in a test have reached the first phase of their life cycle. Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.
KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The precondition is a knowledge requirement that must be true before proceeding (e.g. ServicesStarted for Phone App 1) Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.
KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The temporal delay is a platform-neutral sleep of at least the specified number of seconds Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.
KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The postdelay is a knowledge expression to evaluate after the temporal delay has finished. Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.
KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The application launch creates a heavy weight process to run the user-specific executable Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.
KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The postlaunch phase begins with the evaluation of user-provided knowledge expressions and extends until the application exits or is killed Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.
KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • The postcondition is evaluated after the application has exited. A return value is made available to the user-specified knowledge expression Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.
KATS: Testing Entity Life Cycle Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Each testing entity has eight life cycle phases which flow linearly until completion. Each phase is disabled by default. Barrier Application Launch 5. 1. Precondition Postlaunch 6. 2. • During the exit phase, the testing entity cleans up and ends its life cycle Temporal delay Postcondition 3. 7. Postdelay Exit 4. 8.
KATS: Testing Entity Modeling Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • A GME DSML paradigm is provided for modeling all testing entities: • Visual interface for configuring entities and their life cycles • The modeled entity attributes are generated into an XML file • Recursive includes are allowed to chain entity groups and provide for intricate testing sequences • Provides specialized KATS processes like the KATS Observer, which allows for knowledge monitoring of active tests
KATS: Disseminating Knowledge Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • KATS uses the MADARA KaRL reasoning engine • Portable reasoning engine built for distributed real-time knowledge services • Microsecond latency on evaluations • Capable of evaluating millions of knowledge rules per second • Built in C++ on top of ACE • Provides network transports for two DDS vendors • Open Splice Community Edition (open source) • NDDS (licensed and closed source)
KATS: Disseminating Knowledge Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • KaRL provides an intuitive multimodal logic • Supports declarative or rule-based expressions • .kats.result == 2 => PhoneApp1.failed = 1 • PhoneApp1.failed && Services.running && (StartTest2 = 1) • Supports local variables (variable starts with a period) which are not disseminated • Global variable values will be disseminated to all entities in the domain when changed • First class support for 264 knowledge values per variable • No available ontologies for non-integer types but can be supported
KATS: Smart Phones and MAML Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • So… what does this have to do with Smart Phones? • MADARA contains a Python library called MAML which allows for instrumenting an Android smart phone from a connected host machine • Uses a combination of Android Monkeyrunner and the Android Debug Bridge • Can also run instrumentation apks (installed programs that manipulate a different installed program) and JUnit tests • The MAML library is open source and may be downloaded separately from the rest of the MADARA source code
KATS: Smart Phones and MAML Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • So… what the heck is Monkeyrunner? • The functional parts of Monkeyrunnerallow for sending KeyEvents (like mimicking a user clicking a button, etc.) to the smart phone • Also allows for taking snapshots of the screen and saving it to disk • Wait… so what does this MADARA MAML thingy do? • Encapsulates the working Monkeyrunner functionality • Fixes the broken parts of the API • Provides mechanisms for directly using the Android Debug Bridge • Provides convenience functions for instrumentinglogcat from Python • Provides a much more user friendly library of functions to send KeyEventsthan Monkeyrunner does
KATS: Smart Phones and MAML Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Okay… but what does MAML have to do with KATS? • Python scripts that use the MAML library may be fitted with the testing entity life cycle of KATS to instrument smart phones • KATS are MAMLs…
KATS: Demo the thing already! Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Why are you looking at this screen? • Our powerpoint instrumentation feature is broken…
KATS: Conclusions Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • KATS features a flexible test entity life cycle model that provides fine-grained sequencing support for automating almost any test, including most that require microsecond precision to replicate race conditions between hosts • The KATS paradigm for GME provides an intuitive interface for configuring all aspects of tests and their sequencing • The KaRL engine provides a dynamic knowledge and reasoning service that is built into appropriate KATS life cycle phases • The MAML Python library provides a extensible interface for instrumentingAndroid Smart Phone applications • One day, these technologies may contribute to a distributed, automated deployment that walks your dog and waters your plants while you are playing with robots at work.
KATS: Conclusions Scenario Overview Modeling Disseminating Smart phones Demo Conclusions • Email • James Edmondson (james.r.edmondson@vanderbilt.edu) • Websites • MADARA project site (http://madara.googlecode.com) • Distributed Reasoner blog (http://distributedreasoner.blogspot.com)