240 likes | 369 Views
EMI 2 Matterhorn roadmap. Bal ázs Kónya (Lund University) EMI Technical Director EGI Technical Forum, Lyon, 20 September 2011. outline. EMI is about software, the mission of the project is to deliver software for DCIs EMI development cycle Development highlights EMI 2 ”Matterhorn”
E N D
EMI 2 Matterhorn roadmap Balázs Kónya (Lund University) EMI Technical Director EGI Technical Forum, Lyon, 20 September 2011
outline EMI is about software, the mission of the project is to deliver software for DCIs • EMI development cycle • Development highlights • EMI 2 ”Matterhorn” • Content • Timeline Without dwelling into the details EGI TF, Lyon
EMI on the DCI landscape DNA1.4 - EMI Roadmap and DCI Collaborations EMI EGI TF, Lyon
First peak has been conquered: EMI 1 Kebnekaise (2104 m), 12 May 2011 Support to be provided until 28.2.2013 EGI TF, Lyon
Next peak to climb: EMI 2 Matterhorn (4448 m), April 2012 EGI TF, Lyon
We have started the journey • We defined the Plan (what to climb, how to go, what to carry) • DNA1.3.2 ”Technical development plan” • We have the necessary tools • build & integration, task tracking • We are on the way EGI TF, Lyon
From requirements to released products EGI TF, Lyon
While climbing to the peak .... • In addition to EMI 2 development, EMI has a large ”base camp” to take care of • Maintenance • bug fixing • Support • issue resolution • The ”base camp” has the highest priority EGI TF, Lyon
Matterhorn: requirements • Sources of requirements • EGI-TCB: formal requests communicated via EGI Tracker • WLCG: informal requests communicated verbally or via email • Users: direct communication with PTs • Handling of requirements • Requirement collection and processing for EMI 2: CLOSED • No major changes in EMI 2 directions • Last minute case-by-case adjustments are possible • Requirement collection for EMI 3: OPEN • Requirement assessment is a continously running process • Everything is recorded and tracked: • https://savannah.cern.ch/task/?group=emi-req • note: requirements from the project proposal phase are not there EGI TF, Lyon
Matterhorn: requirements Endorsed top priority requirements: • WMS should provide better feedback about jobs, automated resubmission of stuck jobs • Release EMI clients on SL5/32 • Standard file locations for logs and temporary files • Publish mw service version • Provide Nagios probes for EMI services • Better error messages (more verbose, informative) • Coherency of command line commands parameters • Coherency of APIs • Support for Debian 6 • Support for SL6 • Ubuntu support at least for EMI clients • CEs should support a set of LRMS • Reduce memory consumption of CREAM EGI TF, Lyon
Matterhorn: requirements and some non-accepted ones: • Fixing known bugs before adding new features • Uniform logging • Improving (automatization) yaim configuration • Unicore GUI improvements EGI TF, Lyon
EMI Workplan (3 years) EGI TF, Lyon
Dev highlights: compute (1/2) • Implementation of the agreed common job management methods (EMI-ES interface) • CREAM, UNICORE, A-REX Computing Elements • All EMI compute clients • Complete GLUE2 support • GLUE2 used in resource discovery, brokering, matchmking • Agreement and support for an EMI compute accounting record (UR) • ARGUS integration of computing elements • Lots of product-level improvements • see next slide as an example EGI TF, Lyon
Dev highlights: compute (2/2) gLite compute PT plans: • CREAM High Availability • Pool of CREAM CEs seen as a single logical CE • (S)GE support for CREAM • WMS Support for DAG jobs without using Condor DAGMan • This will enable the submission to CREAM CEs also for DAG jobs EGI TF, Lyon
Dev highlights: data (1/2) • GLUE2 support in data clients • EMI Data Access Library design and implementation • Storage Element and catalogue synchronization • Will not be released with EMI 2 • Better UNICORE data integration • SRM, LFC, AMGA • NFS 4.1 / pNFS for EMI storage elements • focus on standard protocols and procedures • POSIX file access and integration into SRM (File://) • http/WebDAV/Cloud storage details: Monday Data Workshop presentations EGI TF, Lyon
Dev highlights: data (2/2) What we actually want Google Storage Gs,S3 REST, WebDAV HTTP • Amazon • S3 • CloudFront Gs,S3 REST, WebDAV HTTP FTS 3 ROOT EMI Storage EMI Storage EGI TF, Lyon
Dev highlights: security • Finalization of ”EMI profiles” • Common SAML profile • Common XACML profile • Security Token Service (STS) service • Simplified management of security credentials (AAI) • Authenticates and authorizes users based on security tokens • Transforms the security token into another security token • Implementation based on Shibboleth IDP & OpenWS/SAML v3 (Shib3) • EMI Authentication Library implementation • JAVA, C, C++ APIs • EMI delegation agreement • A delegation solution to replace ”GSI” in compute and data areas Details: Wednesday ”EMI Security” session EGI TF, Lyon
Dev highlight: infra EMI Service Registry • Full utilization of GLUE2 including validation • EMI Service Registry • Delivery of service monitoring via NAGIOS • Each EMI service should come with probes • Modified accounting publishers • DGAS, APEL • Accomodate new records • Utilize messaging EGI TF, Lyon
Dev highlight: common/cross area • Cloud strategy (see next talk) • Consolidation plans for each area • What can be phased out? • Additional platforms • Debian 6 (with an eye on latest Ubuntu) • SL6 • Usability improvements • Command line parameters • Error messages EGI TF, Lyon
Matterhorn: what will be inside the box? • EMI 2 will be largely based on products taken from EMI 1 • EMI 1 contains 54 products from ”AMGA to WMS” • Product updates with possible backward incompatible changes • New products: • STS, Pseudonimity • EMI Service Registry, EMI_datalib, EMI_authlib • Components to be phased out • Not yet decided, consolidation plans are due in October • There’ll be products tagged as ”discontinued” • EMI project is contractually obliged to cut its portfolio • The content will be finalized as part of the release preparation • We are strict on deadlines, delayed products can and will drop out EGI TF, Lyon
Matterhorn: release timeline (1/2) • Development tasks defined and being tracked - July, 2011 • Some task are not yet recorded • BUILD environment ready for all platforms: – August, 2011 • Not fully completed • QA policies and procedures ready – September, 2011 • RC1 (1st code freeze) – 14th Dec, 2011 • Successful build and packages available • All foreseen functionalities should be implemented • RC2 (2nd code freeze) – 1st Feb, 2012 • Bug fixes • Available test reports for in-house deployment of RC1 EGI TF, Lyon
Matterhorn: release timeline (2/2) • RC3 ( 3rd code freeze) - 29th Feb, 2012 • RC2 functionality and config testing, deployment on EMI testbed • RC4 (final release) - 28thMar, 2012 • Full test reports for RC3 • Fully deployed RC3 on the EMI testbed • Only authorized bug fixes (tracked) • Final Release available in testing repository - 30th Mar, 2012 • Complete testing and certification reports available • All tasks “Certified” • Announce EMI 2 (Matterhron) - 26th Apr, 2012 • Preparation, planning for EMI 3 starts – January 2012 EGI TF, Lyon
Matterhorn in a nutshell • The next Major release from the EMI project • Contains appx. 50-60 integrated products for compute, data, security and infra capabilities • Backward incompatibility allowed! • The most development-dense EMI release • Will be available on multiple platforms • Planned release date: 26 April 2012 • Support will run well after the end of EMI project EGI TF, Lyon
Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 EGI TF, Lyon