TeraGrid Information Services

1. TeraGrid Information Services JP Navarro, Lee Liming University of Chicago TeraGrid Architecture Meeting September 20, 2007

2. Grid in what sense? People Interactions Identify available resources and request allocation User Documentation, User Portal, POPS Learn how to use resources and their status User Documentation, Knowledge Base User Portal Conferences and other events Ask for help E-mail Telephone (User) Software Interactions Standard service interfaces (login, move data, run jobs, WAN file-systems, manage data, etc.) Coordinated software (Grid clients, development tools, science workflow, etc.) Coordinated Unix interaction (standard variables, SoftEnv) TeraGrid Architecture Meeting

3. TeraGrid Information Services Vision Provide an Information Services Infrastructure: • Applying grid concepts to information publishing • Have RP/GIG/partner operated information services • Centrally indexing and/or aggregated for discovery • Primarily focused on public information • Primarily accessible thru software interfaces • Using standards based interfaces • Reliable, scalable, and fast • Publishing TeraGrid information • And partner/community information TeraGrid Architecture Meeting

4. TG Information Services IS [NOT] A coordinated way to index and publish public [Tera]Grid information using software interfaces. TeraGrid Architecture Meeting

5. High-Level Components TeraGrid Wide Services Apache 2.0 WS/RESTHTTP GET TeraGrid Repositories Clients Cache TomcatWebMDS WS/SOAP Clients Partners WS MDS4 Resource Provider Services WS/SOAP Clients WS MDS4 TeraGrid Architecture Meeting

6. TeraGrid Wide vs RP Services RP Information Services Content: RP owned and maintained information Data can originate in local systems Infrastructure: 2 scheduling MDS services: authenticated and public (merging) 1 general purpose MDS service TeraGrid Wide Information Services Content: Aggregate/index RP Information Services content Additional central information (TGCDB, GIG operated services, …) Infrastructure: Several redundant servers Information caching (persistence) Several MDS4 services (WS/SOAP) WebMDS/Tomcat, Apache 2.0, … (WS/REST) Content published in: HTML, XHTML/XML, XML, Atom, RSS, … TeraGrid Architecture Meeting

7. Tools WS/* (Tomcat 5.0, Apache 2.0) Benefits Very common web services platform Supports several web service interfaces (including simple) Supports multiple styles like REST, Web 2.0 Can be highly scalable Content Many formats: HTML, XHTML/XML, XML, Atom, RSS, … WS/SOAP (Globus 4.0.5/VDT 1.7.1 MDS4) Benefits Indexing, Trigger Registration, Publish, Subscribe Security/Authorization Robust WSRF interface Content XML WebMDS Benefits XPath support XSLT transforms TeraGrid Architecture Meeting

8. High-Availability Design Clients info.teragrid.org TG wide servers RP/partner services (Patrick Dorn & NCSA NetEng) info.dyn.teragrid.org TeraGrid Dynamic DNS Dynamically direct clients to one or more servers Set by Information Services administrators Changes propagate globally fast (TTL = 15 minutes) … Doesn’t Change Dynamically Changes TeraGrid Architecture Meeting

9. Information Services Users User Documentation (Michael, Diana) User Portal (Maytal & team) Inca (Kate & team) Gateways Database? Peer Grids User Applications info.teragrid.org TeraGrid Architecture Meeting

10. Completed Milestones Infrastructure information services are in production • RP Information Services • TeraGrid Wide Information Services Scheduling information published to User Portal (since Spring) • Scheduler load • Queue contents CTSS 4 capabilities published in publicly (Since August) • Accessed by • User Documentation • Inca system • Content • Which capability kits are available on each resource • What software is available in each kit on each resource • What services are available from each kit on each resource TeraGrid Architecture Meeting

11. Queue Contents in User Portal

12. Resources and Available Kits

13. Where are the GridFTP services?

14. CTSS 4 Kit Capabilities For each kit on each resource • Current support level, and target support level • Development/prototype, Testing/pre-production, Production • Support organization • Inca status URL • Multiple version of a kit with different support levels For each kit software component on each resource • Name, version, how to access it • Multiple versions of a single component For each kit service on each resource • Name, type, version, and Endpoint (contact location) • GSI OpenSSH, GridFTP, SRB servers, GRAM, MDS4 • Multiple services of the same type The coordinated way TeraGrid publishes available CTSS capabilities. TeraGrid Architecture Meeting

15. Open Forward Process - Requirements Requirements Analysis Design & Develop Deploy & Support • What • Use cases (who needs it)? • What type of information (attributes)? • Who owns and provides the data (ownership, source)? • Is any of it already available thru information services (gap analysis)? • Who • Everyone: users, staff, partners • Where • Discussion: tg-cat@teragrid.org • Analysis results: wiki documents currently linked here: • http://www.teragridforum.org/mediawiki/index.php?title=Information_Services#Potential_Future_Content TeraGrid Architecture Meeting

16. Open Forward Process - Develop Requirements Analysis Design & Develop Deploy & Support • What • Register a provider and/or publish data? • Design/adopt a schema? • Develop publishing adaptors, services. • Choose and/or develop query interfaces. • Who • Several groups: information users, owner, consumers, SI facilitated • Where • Discussion: gig-info@teragrid.org TeraGrid Architecture Meeting

17. Open Forward Process - Deploy Requirements Analysis Design & Develop Deploy & Support • What • Deploy adapters or publishing services. • Deploy query interfaces. • Update information services documentation. • Who • Several groups: information providers, RPs, SI • Where • Discussion: software-wg@teragrid.org, gateways, docgroup, … TeraGrid Architecture Meeting

18. Current Activity Areas Deploy & Support Upgrade/expand/merge scheduling information services Part of WS GRAM 4.0.5 upgrade Doesn’t address Scheduling WG requirements yet Implement 99.5% availability TeraGrid wide information services Improve information services documentation Design & Develop Publish TGCDB resource and organization information Prototyping REST/Web 2.0 interfaces: Publish HTML (staff/developer views), XHTML/XML, Atom, RSS, etc. Prototype universal command line client Requirements Analysis HPC (non-CTSS) software Resource hardware specifications Scheduling information (BQP, Scheduling WG) Data movement information (Data WG) TeraGrid Architecture Meeting

19. [Not so] Farfetched Possibilities Gateways User Services gateway database information Gateway published, capabilities, software, and services Data collections User Services data collections database information Data collections access method, service Endpoint, paths Community software areas Which resources have each CSA What software is available in each CSA, how to access it Community accounts Which resources is each account active on Outages Planned and unplanned outage information (gateways could use) Resource Provider Policies Peer grids/interoperability Resources, services available on peer grids (OSG, EGEE, …) ……. TeraGrid Architecture Meeting

20. More Information Discuss Information Services content, requirements, and design: E-mail list tg-cat@teragrid.org View current Information Services content User Portal (scheduler load & queue contents): https://portal.teragrid.org:443/gridsphere/gridsphere?cid=resources User Documentation (CTSS 4 kits, software, services): http://www.teragrid.org/userinfo/software/ctss.php Staff and developer WebMDS views http://www.teragridforum.org/mediawiki/index.php?title=WebMDS_Views Useful URLs: Will link to other areas: http://info.teragrid.org/ Information Services activities, plan, etc, http://www.teragridforum.org/mediawiki/index.php?title=Information_Services TeraGrid Architecture Meeting