1 / 26

ITPA/IMAGE conference 7-10 May 2007

ITPA/IMAGE conference 7-10 May 2007.

denton-mercado
Download Presentation

ITPA/IMAGE conference 7-10 May 2007

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ITPA/IMAGE conference 7-10 May 2007 Software and Hardware Infrastructure for the ITMB.Guillerminet, on behalf of the ITM & ISIP teams(P Strand, F Imbeaux, G Huysmans, J Lister, G Manduchi, F Iannone, M Airaj, P Huynh …) and others(Y Smirani, V Bergeaud, R Metery, D Drouin, P Micoud, C Leroy …) ITPA/IMAGE 7-10 May 2007

  2. Software & hardware infrastructure for the ITM • ISIP belongs to the Integrated Tokamak Modelling project • In charge of the infrastructure (soft + hard) for the ITM: • Code platform: framework, editor, … • Data handling: data structure, data storage & access, … • Tools: version handling, portal, … • Hardware: gateway, GRID/HPC, … • 3 others talks: • ITM (P Strand) • Data structure (F Imbeaux) • Scientific workflow with KEPLER (M Airaj) • Focus on the software architecture ITPA/IMAGE 7-10 May 2007

  3. Outline • Requirements & challenges • Choices & solutions • Architecture • Data structure • Data access & storage: UAL • Code structure • Intensive computing: GRID/HPC ITPA/IMAGE 7-10 May 2007

  4. Outline • Requirements & challenges • Choices & solutions • Architecture • Data structure • Data access & storage: UAL • Code structure • Intensive computing: GRID/HPC ITPA/IMAGE 7-10 May 2007

  5. Requirements & challenges • Long lasting project: • O.S. will change • New languages • Framework could disappear • Legacy software and language mixing: • Codes written in F77, F95, 2003, C, C++, Java, Matlab, … • Depends on the Fortran compiler • Local/remote computing: • Scheduling (start/suspend/resume/abort …) • Naming, discovery tools • Fast communication: • how to deal with all the above (big/little endian, language mixing, …) • Component: • Standard? • CCA? BABEL • XML: • Binary XML • Fast XML ITPA/IMAGE 7-10 May 2007

  6. Outline • Requirements & challenges • Choices & solutions • Architecture • Data structure • Data access & storage: UAL • Code structure • Intensive computing: GRID/HPC ITPA/IMAGE 7-10 May 2007

  7. Choices & solutions • Architecture in layers • Not an universal solution for any language to any language but valid for any identified data: XML schemas • Standard code description: WSDL • Binary data transfer: UAL • Remote computing: Web Services • Intensive computing: GRID, DEISA ITPA/IMAGE 7-10 May 2007

  8. Outline • Requirements & challenges • Choices & solutions • Architecture • Data structure • Data access & storage: UAL • Code structure • Intensive computing: GRID/HPC ITPA/IMAGE 7-10 May 2007

  9. Architecture • Gateway • Cluster 32-64 CPUs (16GB), 10-100TB, 10Gb/s • UI: access to GRID EGEE • Web portal: • Single sign-on, authentication, security, user’s profiles • Access to ITM applications • Catalogues and simulation storage ITPA/IMAGE 7-10 May 2007

  10. Architecture (cont’d) • Applications: • Workflow engine (KEPLER): • running • Simulation editor/director: • Writing the requirements • Delivered at 4Q07 • Data mining tool • SCILAB: post-processing tool • Work in progress • Control & monitoring tools: • Cluster tools • GRID tools • On-line data visualization: • 2D/3D actors in KEPLER (VTK based) • Visit tool (gateway + GRID) ITPA/IMAGE 7-10 May 2007

  11. Outline • Requirements & challenges • Choices & solutions • Architecture • Data structure • Data access & storage: UAL • Code structure • Intensive computing: GRID/HPC ITPA/IMAGE 7-10 May 2007

  12. Data structure • Not an universal solution for any to any code but valid for any identified data: XML schemas • DATA “equilibrium” • Description: Automatic generation: Xsd2f, … • Structure: • F95 version • 3816 lines • 9MB • No need to know XML ITPA/IMAGE 7-10 May 2007

  13. Data structure (cont’d) • XML schemas • Generate automatically the include files for C, Fortran, Java, Matlab, … • Use it in your code: #include …, import … • Fill the structure through the provided API (UAL): xsd2f, xsd2c xsd2m No XML knowledge needed ITPA/IMAGE 7-10 May 2007

  14. Outline • Requirements & challenges • Choices & solutions • Architecture • Data structure • Data access & storage: UAL • Code structure • Intensive computing: GRID/HPC ITPA/IMAGE 7-10 May 2007

  15. Data access: UAL • Hidden data storage: • MDS+ • HDF5 • Commercial database • Message oriented Middleware (Control Host) • APIs for “any” language: • C/C++ • Fortran 77,95,2003 • Java • Matlab • Fast communication • Binary • In-memory, pointer access • Access: ITM library (UAL): • euitmopen() • euitmget_equilibria() • euitmput_equilibria() • euitmclose() ITPA/IMAGE 7-10 May 2007

  16. UAL as a layer • Communication between codes through the UAL • In-memory data server  whiteboard • Up-to-date data • Time-stamped data • Independent of the framework codes EFIT, … codes HELENA, … codes MISHKA, … UAL data server ITM + private JET, … Exp. data In-memory ITPA/IMAGE 7-10 May 2007

  17. display UAL: MoM implementation Publish-subscribe server Based on a Message Middleware: ControlHost (A.Maslennikov) Remote web server Soloviev “equilibrium” data ~9MB in-memory data server Helena KEPLER • Advantages: • distributed • fast (binary data transfer) • Language agnostic Mishka arguments of the WS ITPA/IMAGE 7-10 May 2007

  18. Outline • Requirements & challenges • Choices & solutions • Architecture • Data structure • Data access & storage: UAL • Code structure • Intensive computing: GRID/HPC ITPA/IMAGE 7-10 May 2007

  19. initialize() Iterations wrapUp() prefire fire() postfire() Code Structure • Independent of the framework: • Able to change it later (layer) • Tools to integrate it (one click option) • Data access through the UAL: • Independent of the storage • The framework does not know our data • Tools to integrate it in the framework: • Local (JNI based tool) • Remote (Web Service based tool) • Structure: No WS knowledge needed ITPA/IMAGE 7-10 May 2007

  20. Code Structure (cont’d) • subroutine initialize() • call euITMopen(‘…’) • return • end • subroutine fire (variables) • call euITMget(‘type’, in) • call code(in, out) • call euITMput(‘type’, out) • return(variables) • end • subroutine wrapup () • call euITMclose(…) • return • end • subroutine code(data in, data out) • … • compute • … • return • end UAL KEPLER Initial code • program test-code • read file => fill in • call code(in, out) • read out => write out • end • subroutine code(data in, data out) • … • compute • … • return • end No change ITPA/IMAGE 7-10 May 2007

  21. Framework: workflow engine • Based on KEPLER • =>graphical tool • =>advanced models of computation • =>directors • =>hierarchical design Run control buttons Scheduler: time behavior Catalogues ITPA/IMAGE 7-10 May 2007

  22. Plot 2D Plot 3D in-memory Data server SOLOVIEV HELENA MISHKA ITPA/IMAGE 7-10 May 2007

  23. Outline • Requirements & challenges • Choices & solutions • Architecture • Data structure • Data access & storage: UAL • Code structure • Intensive computing: GRID/HPC ITPA/IMAGE 7-10 May 2007

  24. GRID-HPC • Gateway • Local codes (through UAL) • 64 CPUs • Computers from the associations • Remote codes (through WS + UAL) • GRID-EGEE Fusion V.O. • Distributed computing • 10K CPUs • HPC-DEISA • Parallel computing • Network of super-computers • GLOBUS • =>C/C++ library • =>Java library in development Euforia?? ITPA/IMAGE 7-10 May 2007

  25. GRID-EGEE Fusion V.O. • GLOBUS • =>C/C++ library • =>Java library in development • GLITE • =>Java library in development • ITM • =>Codes as WS • =>UAL data access Job transfer via FTP Data transfer via FTP Job scheduling via JCL Proxy ITPA/IMAGE 7-10 May 2007

  26. Work plan 2007: development • KEPLER prototype: workflows, visualization 2D/3D, local/remote codes (with graphical tools) • Web portal requirements • User interface requirements (Simulation editor) • UAL C/C++/Fortran APIs (MDS+) • Gateway • User interface delivered (simulation editor) • Repositories (codes, Kepler, …) • GRID-EGEE Glite in Kepler 2008: production • Enhancements: Kepler, User interface • Data mining tools • Web portal • GRID-EGEE ITM • GRID-DEISA July 07 December 07 July 08 December 08 ITPA/IMAGE 7-10 May 2007

More Related