1 / 16

The Virtual Data Toolkit & OSG Software

The Virtual Data Toolkit & OSG Software. Alain Roy. OSG Software Coordinator. Condor Team. University of Wisconsin-Madison. Review: What is the VDT?. A collection of software Grid software (Condor, Globus and lots more) Virtual Data System (Origin of the name “VDT”) Utilities

may-bartlett
Download Presentation

The Virtual Data Toolkit & OSG Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Virtual Data Toolkit& OSG Software Alain Roy OSG Software Coordinator Condor Team University of Wisconsin-Madison

  2. Review: What is the VDT? • A collection of software • Grid software (Condor, Globus and lots more) • Virtual Data System (Origin of the name “VDT”) • Utilities • An easy installation • Goal: Push a button, everything just works • Two methods: • Pacman: installs and configures it all • RPM: installs some of the software, no configuration • A support infrastructure The State of the VDT

  3. Who makes the VDT? • The VDT is a product of Open Science Grid (OSG) • VDT is used on all OSG grid sites • OSG is new, but VDT has been around since 2002 • Originally, VDT was a product of the GriPhyN/iVDGL • VDT was used on all Grid2003 sites The State of the VDT

  4. Who makes the VDT today? 1 Mastermind + 3 FTEs Miron Livny Alain Roy Expanding with new funding Tim Cartwright Andy Pavlo The State of the VDT

  5. Why have the VDT? • Everyone could download the software from the providers… • But the VDT: • Figures out dependencies between software • Works with providers for bug fixes • Provides automatic configuration • Packages it • Tests everything on thirteen platforms (and growing) The State of the VDT

  6. We pre-built binaries for each of these Example: VOMS • VOMS can authorize people in a VO • VOMS has a web interface • We: • Install Tomcat • Install Apache • Built with Globus SSL • Patched so GSI pass-through to Apache works • Install VOMS • Install VOMS Admin • Install Perl modules needed by VOMS Admin • Install MySQL and set up database (with command-line tool) • Configure all software • Configure rotation of log files The State of the VDT

  7. Example: Security Update • Last week, we got security update from Globus • We decided to patch VDT 1.2.x, 1.3.9, 1.3.10, and 1.3.11 • We need to build the updated binaries three times on about six platforms • We need to coordinate creation of patch for Globus 2.4.3 • We need to patch the Globus updates with our patches • We need to take subset of Globus updates • We need to package an update that can be reversed, if there are problems. The State of the VDT

  8. How much software? The State of the VDT

  9. Current Release: VDT 1.3.11 • Basis for OSG 0.5.0 • Adds some of requests for OSG 0.6: • Globus 4.0.2 with TeraGrid modifications • SRM Clients • Squid • Wget • New versions of CEMon and VOMS • Support for Fedora Core 4 on x86-64 The State of the VDT

  10. VDT 1.3.12: In Progress • Likely to include: • Gratia (for accounting) • BDII (including OpenLDAP) • glexec • Edge Services Framework • TclGlobus • Improvements to upgrading the VDT • A slew of bug fixes and smaller enhancements • More at: http://vdt.cs.wisc.edu/future.html • Final decisions about content during this meeting The State of the VDT

  11. VDT Stack => OSG Stack Input from stakeholders and OSG directors VDT Release OSG Integration Testbed Release OSG Production Release The State of the VDT

  12. … Making a VDT release • Requires at least one week of testing on VDT testbed • Run nightly tests on each supported platform until they are clean The State of the VDT

  13. … Making Int. Testbed Release • Add software (mostly config) • Package subsets of VDT software • Compute Element • Storage Element • Worker-node Client • Submit-host Client • … • Deploy on testbed • Test services • Test middleware interoperability • Test functionality and scalability • Application validation The State of the VDT

  14. Challenges we face • How should we smoothly update a production service? • In-place vs. on-the-side • Preserve old configuration while making big changes. • As easy as we try to make it, it still takes hours to fully install and set up from scratch • How do we support more platforms? • It’s a struggle to keep up with the onslaught of Linux distributions • AIX? Mac OS X? Solaris? Fedora Core 6 Gentoo Fedora Core 5 Fedora Core 4 RHEL 3 BCCD Debian Fedora Core 3 RHEL 4 The State of the VDT RHEL 5

  15. More challenges • Distributed creation of VDT • Now we have funding for people to add support for new storage resource management systems (d-Cache, DRM, etc) • These people are not at UW-Madison • Improving testing • We care about interactions between the software: “When using a VOMS proxy with Condor-G, can we run a GT4 job with GridFTP transfer, keeping the proxy in MyProxy, while using PBS as the backend batch system…” • Some people want native packaging formats • RPM • Deb The State of the VDT

  16. Where do you learn more? • http://vdt.cs.wisc.edu • Support: • Alain Roy: roy@cs.wisc.edu • Miron Livny: miron@cs.wisc.edu • Official Support: vdt-support@ivdgl.org The State of the VDT

More Related