1 / 23

Grid-Brick Event Processing Framework in GEPS

Grid-Brick Event Processing Framework in GEPS. CHEP 03 – La Jolla, California. A. Amorim, P. Trezentos, N. Almeida, H. Fei, L.Pedro, J.Villate, H.Wolters. luis.pedro@cern.ch. FCUL - Lisbon. Outline. Introduction Architecture Overview Data Flow How do we do it Action!

rochelle
Download Presentation

Grid-Brick Event Processing Framework in GEPS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid-Brick Event Processing Framework in GEPS CHEP 03 – La Jolla, California A. Amorim, P. Trezentos, N. Almeida, H. Fei, L.Pedro, J.Villate, H.Wolters

  2. luis.pedro@cern.ch FCUL - Lisbon Outline • Introduction • Architecture Overview • Data Flow • How do we do it • Action! • Vantages and disadvantages • On-going and future work • Summary

  3. luis.pedro@cern.ch FCUL - Lisbon Introduction • What is GEPS • Grid-based Event Processing System • Developed on top of Globus • Provides web-based access to grid computing environment for event processing • Developed by GridPT working group

  4. luis.pedro@cern.ch FCUL - Lisbon Introduction • Main Idea Do NOT move data! Each node stores and is reponsible by a subset of the whole data…

  5. High speed Disks Data Center luis.pedro@cern.ch FCUL - Lisbon Introduction • The usual way

  6. Meta-data catalogue User terminal User terminal User terminal luis.pedro@cern.ch FCUL - Lisbon Introduction • Our way Job submit server ………..

  7. luis.pedro@cern.ch FCUL - Lisbon Architecture Overview • User submits a query through a interface to the Job Submit Server (JSS). • Job submitted information will be stored in the Metadata Catalog.

  8. luis.pedro@cern.ch FCUL - Lisbon Architecture Overview • The job is submitted to the grid nodes using Globus API functions. • All the nodes query their own information and retrieve a result.

  9. luis.pedro@cern.ch FCUL - Lisbon Architecture Overview • JSS receives result from the Grid nodes and produces a final one • User can download or consult the final result • User can also visualize the state of the job in each Grid node

  10. luis.pedro@cern.ch FCUL - Lisbon Architecture Overview

  11. Meta-data catalogue Brick Grid node luis.pedro@cern.ch FCUL - Lisbon Data-flow JSS B R O K E R User terminal (PHP interface)

  12. luis.pedro@cern.ch FCUL - Lisbon How do we do it: • Technologies used • Globus • PgSQL • LDAP • PHP • ROOT

  13. luis.pedro@cern.ch FCUL - Lisbon How do we do it: • Relevant Features • Globus • Toolkit that provide GRID API functions • PgSQL • Meta-data catalogue implementation • LDAP • Query Grid node information • PHP • Web interface

  14. luis.pedro@cern.ch FCUL - Lisbon Action! • The human interface Main Page Submit a Job GREED info Job status

  15. luis.pedro@cern.ch FCUL - Lisbon Action! • Enabling ROOT Queries • The job is submitted to the grid nodes • All the nodes query their own information with ROOT and retrieve a ROOT file with a TTree • JSS receives the ROOT files and produces a final ROOT file with the result of the query • User can download or consult the final file because it is a TTree.

  16. luis.pedro@cern.ch FCUL - Lisbon Action! • Enabling ROOT Queries (cont) • Stores the information in each node using TTree’s • Filter the information in each node and retrieve a result file that include a TTree • Join all the result files in the Job Submit Server using a TChain and produces a final TTree that is the query result • View the final result file with a TBrowser or with Carrot

  17. luis.pedro@cern.ch FCUL - Lisbon Action! • Enabling ROOT Queries (reading ROOT files)

  18. luis.pedro@cern.ch FCUL - Lisbon Action! • Enabling ROOT Queries (reading ROOT files)

  19. luis.pedro@cern.ch FCUL - Lisbon Action! • Special Features used • ROOT • TObject • TTree • CINT • Filtering data from TTree • ROOT I/O • TChain • Carrot • Browsing ROOT files • Histograming variables

  20. luis.pedro@cern.ch FCUL - Lisbon Vantages and disavantages • Vantages • Commodity Data Storage • Huge Scalability (400 GB/node) • Granularity • Disadvantages • Load balancing • Suitable storage policy • Fault tolerance • Data replication or Backup

  21. luis.pedro@cern.ch FCUL - Lisbon On-going and future work • Error handling and fault-tolerance • Recover mechanisms for each node • Create a redundancy mechanism to recover from a malfunction in the nodes • Develop a storage mechanism to submit more work to the best nodes • Load balancing • Provide to user several interfaces to submit work

  22. luis.pedro@cern.ch FCUL - Lisbon Summary • A different approach is being developed • There is already a real prototype working • Some (good) results have been achieved • A lot of work already done….. • But …. Still a lot of work to do!

  23. luis.pedro@cern.ch FCUL - Lisbon Acknowledgments • Thank you to those who are developing and participating in this project • A. Amorim (Antonio.Amorim@fc.ul.pt) • P. Trezentos (Paulo.Trezentos@iscte.pt) • N. Almeida (Nuno.Almeida@iscte.pt) • H. Fei (hnfi@iscte.pt) • L.Pedro (Luis.Pedro@cern.ch) • J.Villate (villate@fe.up.pt) • H.Wolters (helmut@lip.pt) • Keep the good work Thanks for hearing me! 

More Related