1 / 24

EMI Matterhorn Roadmap 2011 Lyon: Software Development Highlights

Explore the EMI project roadmap "Matterhorn," detailing software development highlights and future plans for EMI 2 and 3. Discover the key requirements, work plan overview, and development highlights in compute, data, security, and infrastructure domains.

lauril
Download Presentation

EMI Matterhorn Roadmap 2011 Lyon: Software Development Highlights

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EMI 2 Matterhorn roadmap Balázs Kónya (Lund University) EMI Technical Director EGI Technical Forum, Lyon, 20 September 2011

  2. outline EMI is about software, the mission of the project is to deliver software for DCIs • EMI development cycle • Development highlights • EMI 2 ”Matterhorn” • Content • Timeline Without dwelling into the details EGI TF, Lyon

  3. EMI on the DCI landscape DNA1.4 - EMI Roadmap and DCI Collaborations EMI EGI TF, Lyon

  4. First peak has been conquered: EMI 1 Kebnekaise (2104 m), 12 May 2011 Support to be provided until 28.2.2013 EGI TF, Lyon

  5. Next peak to climb: EMI 2 Matterhorn (4448 m), April 2012 EGI TF, Lyon

  6. We have started the journey • We defined the Plan (what to climb, how to go, what to carry) • DNA1.3.2 ”Technical development plan” • We have the necessary tools • build & integration, task tracking • We are on the way EGI TF, Lyon

  7. From requirements to released products EGI TF, Lyon

  8. While climbing to the peak .... • In addition to EMI 2 development, EMI has a large ”base camp” to take care of • Maintenance • bug fixing • Support • issue resolution • The ”base camp” has the highest priority EGI TF, Lyon

  9. Matterhorn: requirements • Sources of requirements • EGI-TCB: formal requests communicated via EGI Tracker • WLCG: informal requests communicated verbally or via email • Users: direct communication with PTs • Handling of requirements • Requirement collection and processing for EMI 2: CLOSED • No major changes in EMI 2 directions • Last minute case-by-case adjustments are possible • Requirement collection for EMI 3: OPEN • Requirement assessment is a continously running process • Everything is recorded and tracked: • https://savannah.cern.ch/task/?group=emi-req • note: requirements from the project proposal phase are not there EGI TF, Lyon

  10. Matterhorn: requirements Endorsed top priority requirements: • WMS should provide better feedback about jobs, automated resubmission of stuck jobs • Release EMI clients on SL5/32 • Standard file locations for logs and temporary files • Publish mw service version • Provide Nagios probes for EMI services • Better error messages (more verbose, informative) • Coherency of command line commands parameters • Coherency of APIs • Support for Debian 6 • Support for SL6 • Ubuntu support at least for EMI clients • CEs should support a set of LRMS • Reduce memory consumption of CREAM EGI TF, Lyon

  11. Matterhorn: requirements and some non-accepted ones: • Fixing known bugs before adding new features • Uniform logging • Improving (automatization) yaim configuration • Unicore GUI improvements EGI TF, Lyon

  12. EMI Workplan (3 years) EGI TF, Lyon

  13. Dev highlights: compute (1/2) • Implementation of the agreed common job management methods (EMI-ES interface) • CREAM, UNICORE, A-REX Computing Elements • All EMI compute clients • Complete GLUE2 support • GLUE2 used in resource discovery, brokering, matchmking • Agreement and support for an EMI compute accounting record (UR) • ARGUS integration of computing elements • Lots of product-level improvements • see next slide as an example EGI TF, Lyon

  14. Dev highlights: compute (2/2) gLite compute PT plans: • CREAM High Availability • Pool of CREAM CEs seen as a single logical CE • (S)GE support for CREAM • WMS Support for DAG jobs without using Condor DAGMan • This will enable the submission to CREAM CEs also for DAG jobs EGI TF, Lyon

  15. Dev highlights: data (1/2) • GLUE2 support in data clients • EMI Data Access Library design and implementation • Storage Element and catalogue synchronization • Will not be released with EMI 2 • Better UNICORE data integration • SRM, LFC, AMGA • NFS 4.1 / pNFS for EMI storage elements • focus on standard protocols and procedures • POSIX file access and integration into SRM (File://) • http/WebDAV/Cloud storage details: Monday Data Workshop presentations EGI TF, Lyon

  16. Dev highlights: data (2/2) What we actually want Google Storage Gs,S3 REST, WebDAV HTTP • Amazon • S3 • CloudFront Gs,S3 REST, WebDAV HTTP FTS 3 ROOT EMI Storage EMI Storage EGI TF, Lyon

  17. Dev highlights: security • Finalization of ”EMI profiles” • Common SAML profile • Common XACML profile • Security Token Service (STS) service • Simplified management of security credentials (AAI) • Authenticates and authorizes users based on security tokens • Transforms the security token into another security token • Implementation based on Shibboleth IDP & OpenWS/SAML v3 (Shib3) • EMI Authentication Library implementation • JAVA, C, C++ APIs • EMI delegation agreement • A delegation solution to replace ”GSI” in compute and data areas Details: Wednesday ”EMI Security” session EGI TF, Lyon

  18. Dev highlight: infra EMI Service Registry • Full utilization of GLUE2 including validation • EMI Service Registry • Delivery of service monitoring via NAGIOS • Each EMI service should come with probes • Modified accounting publishers • DGAS, APEL • Accomodate new records • Utilize messaging EGI TF, Lyon

  19. Dev highlight: common/cross area • Cloud strategy (see next talk) • Consolidation plans for each area • What can be phased out? • Additional platforms • Debian 6 (with an eye on latest Ubuntu) • SL6 • Usability improvements • Command line parameters • Error messages EGI TF, Lyon

  20. Matterhorn: what will be inside the box? • EMI 2 will be largely based on products taken from EMI 1 • EMI 1 contains 54 products from ”AMGA to WMS” • Product updates with possible backward incompatible changes • New products: • STS, Pseudonimity • EMI Service Registry, EMI_datalib, EMI_authlib • Components to be phased out • Not yet decided, consolidation plans are due in October • There’ll be products tagged as ”discontinued” • EMI project is contractually obliged to cut its portfolio • The content will be finalized as part of the release preparation • We are strict on deadlines, delayed products can and will drop out EGI TF, Lyon

  21. Matterhorn: release timeline (1/2) • Development tasks defined and being tracked - July, 2011 • Some task are not yet recorded • BUILD environment ready for all platforms: – August, 2011 • Not fully completed • QA policies and procedures ready – September, 2011 • RC1 (1st code freeze) – 14th Dec, 2011 • Successful build and packages available • All foreseen functionalities should be implemented • RC2 (2nd code freeze) – 1st Feb, 2012 • Bug fixes • Available test reports for in-house deployment of RC1 EGI TF, Lyon

  22. Matterhorn: release timeline (2/2) • RC3 ( 3rd code freeze) - 29th Feb, 2012 • RC2 functionality and config testing, deployment on EMI testbed • RC4 (final release) - 28thMar, 2012 • Full test reports for RC3 • Fully deployed RC3 on the EMI testbed • Only authorized bug fixes (tracked) • Final Release available in testing repository - 30th Mar, 2012 • Complete testing and certification reports available • All tasks “Certified” • Announce EMI 2 (Matterhron) - 26th Apr, 2012 • Preparation, planning for EMI 3 starts – January 2012 EGI TF, Lyon

  23. Matterhorn in a nutshell • The next Major release from the EMI project • Contains appx. 50-60 integrated products for compute, data, security and infra capabilities • Backward incompatibility allowed! • The most development-dense EMI release • Will be available on multiple platforms • Planned release date: 26 April 2012 • Support will run well after the end of EMI project EGI TF, Lyon

  24. Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 EGI TF, Lyon

More Related