1 / 19

Testing & test writing SA3 All Hands Meeting CERN

Testing & test writing SA3 All Hands Meeting CERN. Andreas Unterkircher CERN Grid Deployment. Content. Observations from EGEE II Directions for EGEE III Testing as part of patch certification Test infrastructure and frameworks Test writing. Observations from last year.

brady-baird
Download Presentation

Testing & test writing SA3 All Hands Meeting CERN

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Testing & test writingSA3 All Hands Meeting CERN Andreas Unterkircher CERN Grid Deployment

  2. Content • Observations from EGEE II • Directions for EGEE III • Testing as part of patch certification • Test infrastructure and frameworks • Test writing SA3 all hands meeting

  3. Observations from last year • We have a quite extensive list of tests now • https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests • Test development centered around functionality tests • Stress tests are mainly being done manually • Focus on the test framework (SAM) was too strong and distracted from writing tests • Individual testing through SAM is difficult (registration of machines etc.) • Testing is not well integrated into patch certification (lacking trace of testing in the patch) • Virtualization became indispensible at CERN SA3 all hands meeting

  4. Directions for EGEE III Make/keep tests independent from a test framework For the framework we follow production, i.e. SAM will be replaced by Nagios The framework will be used to ensure that our core testbed is working Enabling the framework to do testing for partner patch certification is of lower priority Development of regression tests If possible move parts of testing into ETICS We are not concerned with unit tests (they should be written by the developers) SA3 all hands meeting

  5. Testing as part of patch certification • Testing necessary for a particular patch • Functionality tests • Regression tests • Tests for attached bugs • A list of functionality tests and regression tests that have to be executed is available (and evolving). • There must be a comment about the outcome of the tests in the patch. The results must be available (pasted, attached, link) • If possible verification of a bug should result in a regression test for the bug SA3 all hands meeting

  6. Functionality tests • We have a list of mandatory functionality tests per service • https://twiki.cern.ch/twiki/bin/view/EGEE/ServiceCertificationChecklist • The functionality tests can be executed on the command line. We provide command line wrappers to execute the tests that were originally written for SAM (VOMS, LFC, WN/CE – more to come) • Description of all available tests: • https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests SA3 all hands meeting

  7. Regression tests • Simple Bash “framework” • Can be executed on the command line • More information: • https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#Regression_Tests • Currently tests for around 20 bugs are available SA3 all hands meeting

  8. Tests for attached bugs For every attached bug there must be a comment in the patch If possible a regression test should be written for every bug in the patch Depends on the bug classification (cf. my talk at the All Hands Meeting in Dublin, Dec. 07), but many bugs are in this category (more than 50% in Dec. 07) The regression test should be delivered soon after patch certification SA3 all hands meeting

  9. Bug regression test classification • Test can be done on one isolated node • Case 1: installation of rpms, configuration with yaim, valid proxy available if needed • Test needs more nodes • Case 2a: a server (also included: BDII) is set up and needs to be tested via a client API/CLI from another machine (more realistic) • Case 2b: an API needs to be tested and needs a server • Case 2b-1: server needs special setup (version etc.) • Case 2b-2: server can be standard • Case 2c: several nodes are involved (FTS) • Test is difficult • Case 3: web interface tests, security bugs (no info available), complicated workflow, problems that only occur after running the service for a long time etc. SA3 all hands meeting

  10. Test infrastructure and framework • A stable and complete testbed is kept at CERN. This includes a dedicated CA and VOMS server • The CERN testbed is monitored and tested by NAGIOS • Partner bring up the machines necessary for certifying and connect them to the testbed • It is up to the partner on how to provide the machines (virtualization, permanent machines) • Connecting to the testbed should be easy (cf. Louis’ talk) • Partner do the tests manually on the command line and provide the results (link/TWiki, attachment to patch, pasting) SA3 all hands meeting

  11. Test infrastructure and framework Testing of partner sites/machines via Nagios will be investigated Goal: A set of Nagios tests will be run against partner sites/machines whenever they are connected to the testbed. This will not happen immediately as Nagios is currently being introduced. We have to get experience with it. Testing must not be slowed down because of framework issues. Thus we must ensure that tests can always be executed manually. SA3 all hands meeting

  12. ETICS as a test framework • We started to investigate the possibility to do testing within ETICS. Areas of interest are • Deployment tests • Immediate deployment test after a build by a developer: For any set of artifacts produced by a developer (typically in the volatile repository) this test should check if well defined versions (production, pps or cert) of the affected gLite services can be updated with these artifacts. • Regular Deployment test for a complete build SA3 all hands meeting

  13. ETICS as a test framework • We started to investigate the possibility to do testing within ETICS. Areas of interest are: • Simple Regression tests • Tests that can be done on one node that is configured to be part of the certification testbed SA3 all hands meeting

  14. Regression test writing • More information: • https://twiki.cern.ch/twiki/bin/view/LCG/SAMTests#Regression_Tests • Check out from CVS • Regression test for bug 123 = one file “bug123” implementing 3 bash functions: • test_bug123_pre () • test_bug123 () • test_bug123_post () SA3 all hands meeting

  15. Regression test writing ./regTest.sh –tl testlists/mybugs.txt ./ regTest.sh ./bugs/ bug1596 bug22165 … test_bug1596_pre () test_bug1596 () test_bug1596_post () ./config/ commonFunctions.sh config.sh ./testlists/ mybugs.txt mybugs.sh 1596 22165 … VO=dteam SE_DPM=se.cern.ch … SA3 all hands meeting

  16. Test writing – missing tests • New list of missing tests (evolving list): • https://twiki.cern.ch/twiki/bin/view/EGEE/GliteTestMissing • Test should be written to be executed on the command line. Eventual integration into other frameworks (SAM, Nagios, ETICS) will be considered later • AMGA: API tests (when released) • APEL • BLAH: test plugins for different batch systems, test plugins that come with CREAM • CREAM CE: direct submission to CREAM, CLI • dCache: test SRM client that comes with dCache • DPM: several tests currently being developed at CERN but more tests are needed: SA3 all hands meeting

  17. Test writing – missing tests • DPM: • DPNS server via DPNS CLI, DPM server via DPM CLI • Daemons on DPM: gridftp, rfio, xrootd, httpd • DPM APIs (C, Python, PERL) • SRM tests against DPM using dCache SRM client commands • FTS: VOMS support, SRM copy channel (dCache – DPM), Java API • glexec • Information System: enhance GSTAT, lcg-info, lcg-infosites • LFC: Perl API • Logging and Bookkeeping: verify and enhance existing tests SA3 all hands meeting

  18. Test writing – missing tests Medical Data Management MyProxy Regression tests SCAS VOBOX: integrate existing tests SA3 all hands meeting

  19. Test writing • First write a test plan • List which functionality should be tested • List which CLI commands should be tested • List which functions/methods in the API should be tested • Write the tests and check with test plan if everything is covered by some test • Provide documentation for the tests • Tests will be checked in CVS SA3 all hands meeting

More Related