1 / 10

A Distributed Data Architecture

This article discusses the challenges of managing large quantities of data globally and proposes a distributed architecture for efficient data management and search. The architecture is designed to be scalable and robust, allowing for the seamless flow of data across different locations.

stanleyv
Download Presentation

A Distributed Data Architecture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Distributed Data Architecture Mark Jessop University of York

  2. Swans

  3. Grid Enabled Swans London Tokyo Cape Town Mexico City

  4. How Big is that Lake? • Heathrow capped at 36 landings per hour. • If half have 4 engines and half have 2, average aircraft carries 3 engines. • Each engine generates around 1GB of data per flight. • 36 x 3 x 1 = 108GB raw engine data per hour. • Factor in the working day and the rest of the world… • …Terabytes and up!

  5. London Tokyo Cape Town Mexico City Managing the Flow of Water

  6. Plumbing Toolkit • Data Repository • Catalogue • Pattern Match Engine

  7. Pattern Match Engine • Pattern Match Control • Data Extractor/Encoder • AURA Encoder • AURA-G • Back Check

  8. DATA DATA DATA DATA DATA DATA MCAT MCAT MCAT MCAT MCAT MCAT MCAT MCAT DATA DATA DATA Data Repository • SDSC Storage Request Broker. • Manages distributed storage resources. • Meta Data Catalogue. • Many configurations. • Heterogeneous. • Efficient data delivery. • C++ and Java APIs.

  9. MCAT A Distributed Architecture • One node per airport. • Single global MCAT. • Stream engine data. • Global Parallel Search. • Present Results. • Scalable. • Robust.

  10. Summary • Large quantities of data arriving globally. • Distributed architecture for data management and search. • Scalable and Robust.

More Related