1 / 22

Grid-Powered Scientific & Engineering Applications

Grid-Powered Scientific & Engineering Applications. Ho Quoc Thuan. INSTITUTE OF HIGH PERFORMANCE COMPUTING. Agenda. Introduction Grid@IHPC Real-life Applications Future Work. Introduction. Grid computing:

rolf
Download Presentation

Grid-Powered Scientific & Engineering Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid-Powered Scientific & Engineering Applications Ho Quoc Thuan INSTITUTE OF HIGH PERFORMANCE COMPUTING

  2. Agenda • Introduction • Grid@IHPC • Real-life Applications • Future Work

  3. Introduction • Grid computing: • Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource. (From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”) • Benefits: resource sharing • Issues: • Connection • Dynamic environment • Heterogeneity • Security • Management

  4. Grid@IHPC • To Grid-enable scientific computing focusing on end-to-end solutions • Developing Grid computing middleware: • Information service • Execution management • Data management • User interfaces … • Grid-enabling applications: • Parallelize applications • Integrate applications with Grid middleware • Deploy applications • Supply resources and execute applications …

  5. Grid@IHPC Deployment Service Data Grid Workflow Multimedia Grid Information Service GUI Info. Grid Portal Data Management Compute Grid Command Multimedia Management Access Grid APIs Infrastructure – IT Services Execution Management Components Integration Interface

  6. Information Service Site Information Service Site Information Service End users Allow users to discover & select resources that meet their needs. • Capturing/publishing/supporting queries • Large-scale information collection • Able to publish any types of information: • Hardware • Software • Workload ... • Hot plugable sensors VO Information Hub Grid VO … Info Agents & Info Sensors Info Agents & Info Sensors … … Resources Resources Site X Site Y

  7. Execution Management Framework Site IS Site IS Metascheduler Client • Distribute job across VO resources • Integrated with IS for dynamic resource discovery & matching • Integrated with DS for file movement • Assist clients to reserve resources using access token • Allow workload optimization plug-in • Job Execution Service component at resource layer for queue management & multiple job submission • Integration with LRM (SGE, Torque, LSF) • GT4 and DRMAA compliant • Sequential, simultaneous and MPI jobs • Allow hybrid execution modes Client Client dynamic discovery and matching Grid VO Information Hub reservation Execution Service Execution Service LRM Execution Service LRM LRM integration Sensor Sensor Sensor Sensor Sensor Sensor Handle-driven Hybrid Forwarding

  8. Data/Software Management • Automatic indexing, file change notification • Supporting search and application mapping (search-found-process) Monitored Directory Files Files Client Update Metadata Store File Sensors XQuery Metadata Information (XML) Query by Keyword Data Component OGSA-DAI LFN of files with matching keywords Content Store Metadata & Appln mapping info List of PFN XQuery Query with LFN Content Indexing OGSA-DAI MetadataStore File To ApplicationMapping Replica Catalog (LRC) Replica Catalog (RLI) Insert LFN Data Manager Distributed Storage (1) . . . Distributed Storage (N)

  9. Multi-modal GUI XML Form Man Page Automatic Form Generator User editable Command Line Application User Customizations Dynamic Form Renderer Rendered in Runtime

  10. Grid Portal • Remote access to Grid resource via browsers • Submit batch job • Monitor job status • Download data • Secure access

  11. Workflow Editor • Grid-based PSE for Engineering of Material • Target complex scientific workflows • Integrate with Grid middleware

  12. Virtualization & Resource Management Grid Middleware SUN GRID ENGINE MPI VM1 VM2 VM3 VMn … User Workspace Software/VM Repository • Apply virtualization technologies (Xen, VMWare) • Package scientific tools in virtual machine images for different applications (template-based) • Quick deployment • Easy backup/recovery/migrate • OSs as resources Configurations use and accounting customization provisioning vm vm vm vm vm Xen/VMWare Hardware Resources discovery selection deployment adaptation

  13. End-to-End Solution Middleware Resources Interfaces Construct & run applications Capture, monitor, publish information Capture information Upload, download & search data Extract information, transfer data Store data Submit, monitor jobs Schedule jobs Execute jobs View/download results

  14. Real-life Applications • Fire Dynamic Simulation (FDS) • Multiphysics (Femlab and Comsol) • Computational Electromagnetics (GECEM) • Motif Finder (GIS) • Digital Media Grid (Adaptive Enterprise@Singapore)

  15. Grid-enabled Fire Dynamic Simulation Fire & smoke simulation Compute intensive applications Data intensive (output) Metascheduler Portal Execution Service Execution Service LRM Sensor Sensor Sensor Sensor

  16. Grid-enabled Multiphysics license • Grid-enabled commercial software (Femlab, Comsol) • Resource discovery & execution • Different models of sharing: • Sharing license • Sharing licence & computational power • Sharing desktop (1) License Server User jobs licenses License Server Computing Servers User (2) high bandwidth and low latency network (2) licenses desktop License Server Desktop Servers User (3)

  17. Computational Electromagnetics UWS BAE SYSTEMS Geometry data Generate mesh Create geometry Other locations Output Mesh Output Output IHPC CEM simulation Grid-enabled Computational Electromagnetics (GECEM) • Grid infrastructure • Grid middleware • Portal • Remote visualization • Security Partners: • BAE SYSTEMS • HP UK • University of Swansea • Cardiff University

  18. Grid-enabled Motif Finder • Multi-modal GUI • Session management

  19. Digital Media Grid Project(IHPC, IDA, HP, NTU, SMU) Portal LSF Grid Submit job http LSF Meta-scheduler web browser ssh Maya submit ssh JES Submit job SGE IHPC Meta-scheduler SOAP JES IHPC IS animator JES Torque Web service

  20. Future Work • Build a powerful platform to support science & engineering applications – A*STAR Digital Nervous System (ADNS) • Share resources for inter-research institutes across A*STAR • Provide software as utility services

  21. Status Updates 2006 • Information Service: • Multi-domain • P2P • Hot-plugable sensors • Execution Management: • Resource reservation for parallel/interactive applications • Interface to Local Resource Managements: • SGE, LSF, Torque • Data Service: • Reliable data transfer • Transfer/Stream data in parallel • Data management • Multi-modal GUI • Workflow editor • Virtualization technologies • Applications: • 18 GB of data • More than 30 hours of processing time • Windows applications • Start design and implement ADNS 2007-2008 ADNS in action 2005 • Information Service: • Sensors • Data Collection Server • Grid/Web Service • Execution Management: • Execution Service Container can execute: • Sequential jobs • Concurrent jobs • Metascheduler: • Resource selection based on workload and speed. • Grid/Web service • DRMAA standard • Data Service: • Sending data over Grid/Web service protocol • Grid Portal • Applications: • In house codes • Commercial software

  22. Thank You! Q & A

More Related