1 / 21

Technical Overview

Technical Overview. Bal ázs Kónya (Lund University) Technical Director 1 st EMI Periodic Review Brussels, 22 June 2011. Outline. EMI is about software, the mission of the project is to deliver software for DCIs The process: how the development is managed Technical Management roles

lane-levy
Download Presentation

Technical Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Technical Overview Balázs Kónya (Lund University) Technical Director 1st EMI Periodic Review Brussels, 22 June 2011

  2. Outline EMI is about software, the mission of the project is to deliver software for DCIs • The process: how the development is managed • Technical Management roles • Requirements • Development cycle • The outcome: EMI software portfolio • EMI on the DCI landscape • EMI products and their functionality • High-level technical roadmap • Year 2 top priorities 1st EMI Periodic Review

  3. PTB in EMI NA1 Collaborations, exploitation, sustainability Project Management NA1, NA2 Requirements NA1, JRA1 Software & Services Defines Process definition SA1 JRA1 Training NA2 SA2 NA2 In-Reach Dissemination & Out-Reach Certifies Implements Process monitoring JRA1 SA1 Release Candidate 1st EMI Periodic Review

  4. Managing EMI development 1st EMI Periodic Review

  5. Technical Areas and PTs Compute Services Data Services Product Teams Dedicate teams of experts Fully responsible for development, maintenance and unit/system testing ARC CE, UNICORE Services, gLite MPI , gLite Compute, etc Security Services dCache, CERN Data, DGAS, StoRM ,etc Infrastructure Services ARC Container, UNICORE Security, Cesnet Security, Argus, VOMS, etc ARC Infosys, APEL client, DGAS Client, gLite Infosys, EMI Registry, etc 1st EMI Periodic Review

  6. Requirement management • Sources of requirements • EGI-TCB: formal requests communicated via EGI Tracker • WLCG: informal requests communicated verbally or via email • Users: direct communication with PTs • Handling of requirements • After an initial filtering requests are recorded in the EMI Req. Tracker • Large ”background noise”: duplicates, out-of-scope, bugreports, etc... • PTB assesses, categorizes and prioritizes requirements • Endorsed requests are translated into objectives, then to development tasks • Adjusts and/or creates new objectives • Continous process, though requirements may not have an immediate effect on workplan • Special attention to synchronize with EGI/UMD cycles • Everything is recorded, monitored and tracked: • https://savannah.cern.ch/task/?group=emi-req 1st EMI Periodic Review

  7. From requirements to released products 1st EMI Periodic Review

  8. EMI on the DCI landscape DNA1.4 - EMI Roadmap and DCI Collaborations EMI 1st EMI Periodic Review

  9. EMI software portfolio (1/2) • Originates from ARC, dCache, gLite, UNICORE • not a complete union of the four • Initial stack was defined by DNA1.3.1 • consisted of 98 components • ”component table” was the first attempt to define ”what EMI is” • Early consolidation during first year • iterative process via EMI 0 and EMI 1 release preparations • Logical restructuring • Phase out (5 components) • Current software portfolio (DNA1.3.2) • 58 products of different maturity levels • 54 products are released as part of Kebnekaise • Became 51 after further merges • 3 products under certification and 4 in early development • available on SL5/64 1st EMI Periodic Review

  10. EMI software portfolio (2/2) • Consists of services, clients and APIs/libraries • Integrated products that work together • EMI products contribute to the ”dynamic services” DCI layer and offer the following functionalities: • Compute: job execution, parallel job, job scheduling • Data: file access, file transfer, metadata catalogue • Infrastructure: information publication, accounting sensors, information discovery, monitoring probes, messaging backbone • Security: authentication, authorization, attribute authority, attribute authority, credential management 1st EMI Periodic Review

  11. Evolution of the software stack • Need to manage conflicting requirements: • Product and interface stability • Need for consolidation via portfolio cleanup and adoption of common interfaces and libraries • Need to implement new features • Effect on the software stack: • Established code base with little space for ”easy consolidation” • emphasis is on hardening, cleanup • Challenging Product phase out • transition path is necessary • New products for consolidation • common libraries & services • for a transitional period will coexist with pre-EMI solution • Feature requests to be implemented on top of existing products • avoid development from scratch, yet another set of prototypes 1st EMI Periodic Review

  12. High-level Technical Roadmap 1st EMI Periodic Review

  13. Year 1 achievements Find details in JRA1 presentation 1st EMI Periodic Review

  14. Year 2 top technical priorities (1/2) • Compute: • EMI Execution Service: implementation of the agreed common job management methods • GLUE2 support in compute clients • Data: • Client-side GLUE2 support implementation • EMI Data Access Library design and implementation • Storage Element and catalogue synchronization • Security: • Simplified management of security credentials (AAI) • EMI Authentication Library implementation • EMI delegation agreement 1st EMI Periodic Review

  15. Year 2 top technical priorities (2/2) • Infrastructure: • EMI service registry implementation • Cloud strategy • Delivery of service monitoring via NAGIOS • All: • Consolidation plans • Additional platforms • Debian family and SL6 • Usability improvements • Command line parameters • Error messages 1st EMI Periodic Review

  16. Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 1st EMI Periodic Review

  17. Technical Documents • Release planning, management: • https://twiki.cern.ch/twiki/pub/EMI/EMI1Schedule/kebnekaise-v.9.pdf • https://twiki.cern.ch/twiki/pub/EMI/EMI1Updates/EMI_Release_Schedule_v0.4.pdf • Release Tracker: • https://savannah.cern.ch/task/?group=emi-releases • RfC (development task on a PT level): • http://bugzilla.nordugrid.org/show_bug.cgi?id=2394 (glue support for ARC CE) • https://savannah.cern.ch/bugs/?77525 (ARGUS) • https://savannah.cern.ch/bugs/?77527 (ARGUS) • EMI Execution Service inteface: • http://cdsweb.cern.ch/record/1359908/files/EMI-ES-Specification_v1.0.odt • Storage Accounting record: • http://cdsweb.cern.ch/record/1352472/files/StAR-EMI-tech-doc-final.doc 1st EMI Periodic Review

  18. Cloud position (DRAFT!) • Presented at the EGI User Virtualization Workshop • https://www.egi.eu/indico/conferenceDisplay.py?confId=415 • https://www.egi.eu/indico/getFile.py/access?contribId=9&resId=0&materialId=slides&confId=415 1st EMI Periodic Review

  19. Amsterdam slides:What EMI can contribute with • EMI develops/operates in the "service level”, not in the VM level • some of the EMI services can be useful on the VM-level as well: • BDII as the "info service", storing info about VMMs, so on • a generic service registry which will enable the "hooking up" of all the appliances • ARGUS, VOMS as authorization systems • EMI will provide/contribute/build/test grid appliances and “landscape deployments” of EMI services • Landcape deployment example: a complete CE • EMI will utilize VMs under/behind EMI services • e.g. Computing Elements with VM-based nodes • EMI partners have been the driving force behind some of the “cloud-relevant” standards: • Extensions, profiles for GLUE, accounting • Finally, EMI can offer lessons learnt in Grid • Our experience can help not to repeat the same mistakes WP - Presenter Name - EMI First EC Review 22 June 2011

  20. Amsterdam slides: What is NOT in the scope of EMI • Virtual Machine Management layer • Application catalogues • VM Image catalogues • Accounting systems (server side) WP - Presenter Name - EMI First EC Review 22 June 2011

  21. Understanding the near future... EGI User Virtualization Workshop

More Related