Condor & Middleware: NMI & VDT

1. Condor & Middleware:NMI & VDT

2. Condor Team Experience • Condor works on many platforms • Condor build & packaging experience • Condor deployed in many environments

3. Condor: platforms • This isn’t new (Condor 6.0, 1999) • Linux Intel • Solaris (2 versions) • HPUX • IRIX (3 versions) • Digital Unix • We have experience with many platforms • Condor Today (6.6) • Linux Intel (3 versions) • Linux Alpha • Windows • Mac OS X • Solaris (3 versions) • HPUX • IRIX • Digital Unix • Tru64 • AIX

4. Condor: builds & tests • We believe in nightly builds & tests • Running since early 2000 • Runs on every platform • Developers immediately know when something breaks

5. Condor: build pool (subset)

6. Condor: Many environments • Maintain large pool in CS department • Three platforms • Many diverse applications • Have assisted with many pools world-wide: • You’ve seen many of them today • Commercial • Research • Large & Small

7. Question • Can we apply this expertise to building, packaging, and distributing Grid software? • YES • NSF Middleware Initiative (NMI) • Virtual Data Toolkit (VDT)

8. What is NMI? • NMI is a software infrastructure for middleware, especially grid • Many groups contribute software to NMI • Building, packaging, and testing is based here in Madison, with Condor Team members

9. 18 distinct platforms Builds distributed with Condor NMI Builds Sources (CVS) Build & Test Condor pool (30+ computers) Patching … Build Binaries Web GPT src bundles Test

10. Builds benefit from Condor • Fault tolerance • Computer fails—build just runs later • Can use DAGMan for ordered builds • New platform? Just add a computer

11. Condor & NMI • NMI infrastructure builds lots of software • Capable of doing nightly builds and tests • Condor is one of these pieces of software • Helps Condor with nightly builds and regular tests • We use Condor to build Condor (via NMI) We eat our own dog food

12. NMI GRIDS Status • Globus • Condor-G • Network Weather Service • KX509 • GSI OpenSSH • MyProxy • MPICH-G2 • Grid Packaging Tools • Gridconfig • Gridsolve • PyGlobus • UberFTP • NMI Release 4: December 2003 • Six platforms: • RedHat 7.2, 8.0, 9.0 • Solaris 8 • IA64 RedHa • IA64 SuSE Linux • NMI Release 5: coming soon

13. What is the VDT? • A collection of software • NMI software + extras tailored for specific collaborations • Common Grid middleware (Condor, Globus, and more…) • Virtual data software • Utilities • An easy installation & configuration mechanism • Goal: Push a button, everything you need to be a consumer or provider of Grid resources just works • Two methods: • Pacman: installs and configures it all • RPM: installs some of the software, no configuration • A support infrastructure • Coordinate bug fixing • Help desk • Understand community needs and wishes

14. Condor Group Condor/Condor-G DAGMan Fault Tolerant Shell ClassAds Globus Alliance Job submission (GRAM) Information service (MDS) Data transfer (GridFTP) Replica Location (RLS) EDG & LCG Make Gridmap Certificate Revocation List Updater Glue Schema/Info prov. What’s in the VDT? • ISI & UC • Chimera & Pegasus • NCSA • MyProxy • GSI OpenSSH • LBL • PyGlobus • Netlogger • Caltech • MonaLisa • VDT • VDT System Profiler • Configuration software • Others • KX509 (U. Mich.)

15. Will use NMI processes soon VDT Builds NMI VDT Test Sources (CVS) Build Binaries Build & Test Condor pool (30+ computers) Pacman cache Package Patching … RPMs Build Binaries GPT src bundles Build Binaries Test Contributors (VDS, etc.)

16. Recent Success: Grid2003 • VDT was deployed on 27 sites • VDT/Condor Team members contributed expertise and support • VDT provided access to 2000+ CPUs • Condor-G used by most users • Close collaboration promoted understanding

17. Many VDT Customers • GriPhyN collaborators • US-CMS • US-Atlas • LIGO • SDSS • Particle Physics Data Grid • European Data Grid Project • Enabling Grids for e-Science in Europe Project • LHC Computing Grid Project • Alliance Grid Testbed (AGT)

18. Questions?