1 / 16

Building and Testing using Condor

Building and Testing using Condor. Overview. Goals The Condor build and test process Transition to new NMI framework Future plans. Goals. Automated nightly builds and tests of development and stable branches Ability to submit on-demand build and test runs

oleg-mason
Download Presentation

Building and Testing using Condor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building and Testing using Condor

  2. Overview • Goals • The Condor build and test process • Transition to new NMI framework • Future plans

  3. Goals • Automated nightly builds and tests of development and stable branches • Ability to submit on-demand build and test runs • View results of individual runs and overall status.

  4. The build and test process • Builds and tests are submitted as Condor jobs to desired platforms. • For testing, the previously built Condor binaries are installed as a Personal Condor. Tests are compiled and submitted to the Personal Condor to run. • Condor returns the results when everything has finished. • We use Condor to build and test Condor.

  5. This system has worked great for us for some time. Automated nightly build and test runs. Condor works great to run our builds and tests. We’ve learned from our experience and found some things we could improve on. Great results

  6. Pool machines used for Condor and other project builds and tests. Disk space and partitioning not well defined. Not readily apparent which versions of software exist on a system or where to find it. Some lessons learned

  7. Old framework problems • Builds and tests tightly coupled to the framework. • Difficult to make changes to scripts without undesirable side effects. • We created a new build and test system to solve these problems.

  8. The new NMI framework • Development funded by NMI - NSF Middleware Initiative - which includes groups such as the GRIDS Center • NMI project uses Condor and the pool machines for building and testing software releases. • NMI project benefits from new framework

  9. What does the new NMI framework provide? • The same generic framework is used for both builds and tests. • “Glue” scripts are used to connect the builds and tests with the framework. • Build and test scripts are simpler to write.

  10. Example submit file inputs = input_build_runid.src, condor_test.src pre_all = nmi_glue/test/pre_all remote_declare = nmi_glue/test/remote_declare remote_declare_args = all remote_pre = nmi_glue/test/remote_pre remote_task = nmi_glue/test/remote_task remote_post = nmi_glue/test/remote_post prereqs = perl-5.8.5, tar-1.14, patch-2.5.4, m4-1.4.1, make-3.80, gzip-1.2.4 platforms = x86_rh_9,ia64_sles_8

  11. Workflow submit file NMI Pool x86_rh_9 ia64_sles_8 disk archive database

  12. Condor testing benefits • Condor test runs are now submitted per platform as the builds complete. • Condor tests for specific features or long-running tests may be run separately from the nightly runs. • Ability to pass arguments to tests at submit time. • Enhanced data storage of test results.

  13. New NMI pool management • Machines have minimum hardware and disk space requirements. • Any software installed or settings modified after the OS install are managed with cfengine. • Hawkeye used to advertise all of the software and versions installed in the machine ClassAds. • Condor jobs specify which software and version they require in the submit file, and are matched to the correct machine.

  14. The build and test pools The old pool The new pool

  15. Future plans • Collaborators build and test software using our framework/pool • Multi-machine testing capabilities using the Parallel Universe • Automated Condor-G tests

  16. More information • Attend BOF NMI session on Wednesday for more details on the NMI pool setup, framework and glue scripts.

More Related