1 / 29

Enhanced Middleware Architecture for the Remote Instrumentation e-Infrastructure OGF28

Alexey Cheptsov High Performace Computing Center Stuttgart (HLRS), Germany on behalf of the DORII Project March 17, 2010, Munich. Enhanced Middleware Architecture for the Remote Instrumentation e-Infrastructure OGF28. Middleware Architecture for Remote Instrumentation.

morwen
Download Presentation

Enhanced Middleware Architecture for the Remote Instrumentation e-Infrastructure OGF28

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Alexey Cheptsov High Performace Computing Center Stuttgart (HLRS), Germany on behalf of the DORII Project March 17, 2010, Munich Enhanced Middleware Architecture for the Remote Instrumentation e-InfrastructureOGF28

  2. Middleware Architecture for Remote Instrumentation e-Infrastructure for Remote Instrumentation (c) BMW

  3. Middleware Architecture for Remote Instrumentation OUTLINE • Introduction • Requirements to the middleware • DORII middleware architecture • Main components of the architecture • grid services for remote instrumentation • e-Infrastructure front-ends • application development environment • workflow management system • parallel application support • solutions for scientific visualization • User and developer support

  4. Middleware Architecture for Remote Instrumentation Application requirements to e-Infrastructure’s middleware • A full-featured GUI toolkit that enables easy access and discovering of grid resources and services, and serves a rich user collaborative suite • A workflow management and monitoring system for complex scientific experiments • An integrated application development environment, with enhanced debugging and deployment features. • A visualization client for “on-grid” data analysis • Parallel application implementation libraries, with integrated performance analysis possibilities and back-ends for visualization of performance characteristics.

  5. Middleware Architecture for Remote Instrumentation DORII approach DORII Middleware Architecture Grid access and collaboration (GRIDCC) Application Development on Grid (g-Eclipse) Parallel application support (Open MPI) User-level middleware tools Workflow Management (VLab) Visualization and Interactivity (Int.EU.Grid) Core middleware stack (EGEE) Computation Grid Storage Grid Grid Services Remote Instrumentation on Grid (GRIDCC)

  6. Middleware Architecture for Remote Instrumentation DORII Middleware Architecture • IE extends core middleware by interconnection of instruments with the infrastructure • Common Library collects together classes and interfaces for obtaining access to the underlying core services (+ IE) • VCR act as a central user front-end to the e-Infrastructure, facilitating resource browsing, job management etc. • WfMS enables design and management of the application workflow • Visualization and interactivity providing components greatly extend application support within the e-Infrastructure • Open MPI and MPI-Start support parallel MPI applications Main features

  7. Middleware Architecture for Remote Instrumentation Remote Instrumentation Services on the Grid • Represents a virtualization of diverse data sources and provides the traditional Grid with an abstraction of a real instrument/sensor • Provides user clients with an interactive interface to remotely control an instrument. The middleware to interface a sensor/instrument INSTRUMENT ELEMENT (IE)

  8. Middleware Architecture for Remote Instrumentation Access to grid services • offers simple client-side API for accessing core middleware + IE services deployed on the infrastructure • equipped with a single access point for the user-side middleware – a CL client • simplified installation procedure for both client and user sides • utilized by client-side middleware components • currently compatible with gLite Common Library

  9. Middleware Architecture for Remote Instrumentation e-Infrastructure’s GUI toolkit • a full featured front end to the e-infrastructure and underlying services (IE, …) • ready-to-use collaborative environment, offering a set of groupware tools (logbook, chat, wiki) • entry point for further components (workflows, visualization etc.) VCR

  10. Middleware Architecture for Remote Instrumentation Workflows for complex experiments • supports users in defining, managing and monitoring remote instrumentation scenarios • client-server architecture with light-weight client side WfMS

  11. Middleware Architecture for Remote Instrumentation Workflow Management System (WfMS) • User friendly Swing GUI • Creating and editing workflow • One-click submission on Grid • Monitoring workflow execution client - Workflow Editor and Monitor server - Workflow Manager WEB SERVICES • Java and Web Services • Manages workflow's jobs • Common Lib used for Grid interaction • Three layers design (manager, data base, Common Lib)‏

  12. Middleware Architecture for Remote Instrumentation User Frontend/VCR A workflow example e-Infrastructure 3 2 Computing resources GVid Remote Instrument Storage Resources INGV / CINECA Experiment properties OPATM-BFM 1 Data from CINECA IM User area Constant data

  13. Middleware Architecture for Remote Instrumentation The WfMS architecture • Three layers: • workflow management • database • Common Lib (OSGi services)‏ • Comunication editor ↔ workflow management layer and workflow management layer ↔ OSGi services using web services • Database connected with workflow management layer using Hibernate.

  14. Middleware Architecture for Remote Instrumentation Workflow Editor • Create the workflow • VO-driven access to applications • predefined default applications scenarios • Grid browsing (SE, CE, IE) in workflow's data blocks • Run the workflow on the Grid • one-click workflow submission • workflow's logic is managed by WfMS • Job's output download • Monitor execution progress • presentation of state of each workflow's job

  15. Middleware Architecture for Remote Instrumentation Workflow Manager • Workflow execution steering • Reading and traversing of workflow graph • Translating workflow's jobs to JSDLs, IE and data operations (SE)‏ • Authorisation & Authentication • operations on Grid using user's Proxy certificate • MyProxy delegations to user's proxy accuired from VCR • Users session persistence • Operations on Grid • Job submission, operations on IE, job status check, data operations • done via Grid access libraries (gLite: g-Eclipse)‏ • SOA and Web Services

  16. Middleware Architecture for Remote Instrumentation Integration with VCR • User launches Workflow Editor from VCR • VCR prepares information • User works with WF-E • WF-M identifies user with JNLP file and proxy delegation • WF-M monitors if proxy is valid and renews it when needed (MyProxy)‏

  17. Middleware Architecture for Remote Instrumentation Development workbench for the Grid (g-Eclipse) • Grid + Eclipse = g-Eclipse • Integrated, middleware independent Grid enabled workbench tool • Architecture • Implementation layer (middleware specific)‏ • Abstraction layer (standard middleware functionalities)‏ • Eclipse platform (Eclipse UI, resources management, Eclipse extension points)‏

  18. Middleware Architecture for Remote Instrumentation grid middleware supported in g-Eclipse • gLite • Job management (JSDL/JDL, parametric JSDL)‏ • Remote file systems (gsiFTP, SRM, LFC)‏ • VOMS proxy extension, MyProxy support • Application deployment • Glue information system • Instrument Elements support • GRIA (industry Grid middleware): job management, GRIA file store • Globus GT2: job management (RSL), gsiFTP, gLogin • Amazon Computing Cloud: S3 storage

  19. Middleware Architecture for Remote Instrumentation g-Eclipse in DORII • access and development platform that will be used whenever the application developers need an IDE for their sequential or parallel code • an integrated and user-friendly middleware independent environment • extends the native Eclipse IDE for different Grid actors (users, operators, developers) • provides building blocks for construction of Common Library

  20. Middleware Architecture for Remote Instrumentation Solutions for scientific data visualization A practical use case: OPATM-BFM simulation application the size of the produced short-term forecast data: 72 GB

  21. Middleware Architecture for Remote Instrumentation • enables online communication between nodes on the Grid and off the Grid • provides shell functionality for accessing Grid nodes • is a standard Grid job • supports GSS-based encryption • provides redirection of stdin/stdout/stderr, supports traffic forwarding GVid glogin • Visualization is rendered on the remote node and the output • is streamed as MPEG-4 data to the client • For interactivity the user interaction is transferred back to the render node • Data transferred using glogin • Works transparently with OpenGL/GLUT applications, • provides VTK binding

  22. Middleware Architecture for Remote Instrumentation GVid based visualization applet • Implemented converter from NetCDF to VTK data format • VTK plug-in is developed for g-Eclipse IDE

  23. Middleware Architecture for Remote Instrumentation Parallel applications support • The data collected from the instrument are post processed by simulation applications A practical use case: OPATM-BFM simulation application Simulation duration for a standard use case: ~8 hours (64 CPU)

  24. Middleware Architecture for Remote Instrumentation Parallel applications support Open MPI PACX-MPI MPI Start • Production quality middleware • Full MPI-2 standard conformant • Portable and maintainable • Thread safe • Network and process fault tolerant • Enables running parallel applications on a metacomputer consisting of different CEs • Installed on top of the main MPI library • No changes in the application source code are required • Improve starting parallel applications on the Grid • Supports different MPI libraries, file systems, resource management and process launch systems • Easy usable and extendable

  25. Collection of input I /O operations netCDF files … P1 P2 P32 Collection of output Message - passing netCDF files communication Middleware Architecture for Remote Instrumentation Open MPI • improved MPI-IO functionality • support of PNetCDF library • integration with performance analysis tools (Vampir, Valgrind) • enhanced profiling possibilities thanks to PERUSE interface

  26. Middleware Architecture for Remote Instrumentation DORII Releases. DORII@GForge • Source code management system (subversion) • File release system • Bug trackers

  27. Middleware Architecture for Remote Instrumentation User and developer support www.dorii.eu/middleware DORII middleware page dorii-user-support@... user support mailing list Bug and issue trackers

  28. Middleware Architecture for Remote Instrumentation Thank you for attention! Questions

  29. Remote Instrumentation Support on the Grid

More Related