1 / 21

How Things Work

How Things Work. Enterprise Design Patterns and Weather Data Content Distribution Network (aka'The Cube') Topologies. Design Patterns. The 'Gang of Four' (GoF) book. Adapter Facade Proxy Observer. Gateway Model/View/Controller Registry Repository Service Layer. Aggregator

alisa
Download Presentation

How Things Work

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How Things Work Enterprise Design Patterns and Weather Data Content Distribution Network (aka'The Cube') Topologies

  2. Design Patterns The 'Gang of Four' (GoF) book • Adapter • Facade • Proxy • Observer • ... • Gateway • Model/View/Controller • Registry • Repository • Service Layer • ... • Aggregator • Content Filter • Recipient List • Request/Reply • Publish-Subscribe Channel • ... • Common architectural patterns that have emerged over time • Technology-agnostic – Technologies can change (C/C++/Java/Scala?) over time, but patterns remain consistent • Documenting them and naming them provides a common vocabulary for design discussions • Architecture of entire enterprise can often be described in terms of combinations of these patterns

  3. Enterprise Integration Patterns Messaging Channels System Management Message Channel Request-Reply Guaranteed Delivery Wire Tap Message Store Control Bus Data Type Channel Channel Adapter Publish-Subscribe Message Translation Invalid Message Channel Dead Letter Channel Messaging Bridge Message Translator Message Routing Messaging Endpoints Message Construction Envelope Wrapper Message Router Message Aggregator Message Endpoint Event-Driven Consumer Return Address Content Filter Resequencer Message Filter Messaging Gateway Competing Consumers Message Sequence Content Enricher Recipient List Composed Msg Proc Message Dispatcher Messaging Bridge Normalizer Message Splitter Routing Slip Polling Consumer Service Activator

  4. FUSE Message Broker(a.k.a Apache Camel)

  5. WFS/WCS Distribution Node Reference Implementation in EIP Notation WFS/WCS Service Message Filter (Product Type) Content Filter (Spatial/Temporal) WFS/WCS Aggregator Publish-Subscribe Channel Receive Channel List (Data Aggregation) Recipient List (Subscriptions) Asynchronous Data Store • Apache Camel provides a number of EIP-pattern components out of the box, and a general-purpose framework to connect the components together, based on either: • XML configuration files. Simple, but relatively static • Java Domain-Specific Language (DSL). Requires writing Java code, but more flexible

  6. General Data Distribution Topologies and Associated Enterprise Patterns • Centralized • Pros • Management/Monitoring • Predictable Performance • Cons • Scalability • Fault-Tolerance • Long-Haul Comm Costs • Peer-to-Peer • Pros • Scaleability • Fault-Tolerance • Long-Haul Comm Costs? • Cons • Management/Monitoring • Predictable Performance • Hub and Spoke • Pros • Scalability • Fault-Tolerance • Long-Haul Comm Costs • Predictable Performance • Cons • Management/Monitoring • NNEW focus is Hub & Spoke (Distribution Servers, Origin Servers) • Use of Aggregation and/or Dissemination near the network edge reduces total comm requirements • Does Replication have a role ? • What about Load Balancing ?

  7. Notional Multi-Domain Weather DataContent Distribution Network NOAA responsible for public distribution of weather data Majority of weather data used by FAA to be produced by NWS Gateway(s) Gateway(s) Gateway(s) Gateway(s) Weather Data Clients Public Internet Automation Systems FAA Infrastructure NOAA Infrastructure Weather Models FAA-Resident Aviation Weather Processing NWS-Resident Aviation Weather Processing NOAA Weather Sensors FAA Weather Sensors

  8. Cross-Domain CDN Tiers NWS External DMZ's FAA External DMZ's Top Tier Distribution Servers and RegReps Top Tier Distribution Servers and RegReps Edge Tier Distribution Servers Weather Model Provider (Origin Server) • Replication strategy at top-tier for SAS products (fault-tolerance) • Non-critical products could be handled differently (pulled to top-tier from origin server on-demand)

  9. Aggregation helps with the ‘Complex Retrieval’ case SAS Temperature SAS Humidity Dist Server “Give me SAS Temperature AND SAS Humidity at X,Y, Altitude” Aggregation allows this to be a single request

  10. Content Delivery Network Routing Design Goals • See Wiki...

  11. Notional Client/Origin Server/Distribution Server Psuedo-Collaboration Diagram • Each origin server may itself be fault-tolerant • Primary and backupdsf sources considered separate datasets (unique dataset ids) Origin Server Precip (Primary) Origin Server Precip (Backup) wx-primary.noaa.gov:80 wx-backup1.noaa.gov:80 7: Service Request RegReps Top Tier Distribution Servers wx1.backbone.faa.gov:80 wx2.backbone.faa.gov:80 wx3.backbone.faa.gov:80 6: Service Request 1: Find services for SAS 'Precip' datasets Edge Tier Distribution Servers 4: Service Request 2: wx-primary.noaa.gov:80 wx-backup1.noaa.gov:80 • Origin server address passed along with service request to proxy (ala HTTP Caching Proxies) • Routing controlled independently at each tier • Top-tier distribution servers notionally wouldn't need proxy lookups • Distributed proxy information (file-based implementation initially) can be managed regrep in future • For 2010, top-tier servers could be set up to subscribe to origin server data without waiting for incoming requests 5: Resolve proxy services 3: Resolve proxy services Proxy service address resolver

  12. Proxy-Cache

  13. Proxy-Cache-3/n tier

  14. Distribution Architecture Brainstorming • QoS includes fault tolerance, which is implicit on previous slide • QoS also addresses latency • Need to 'impedance match design to current/planned FTI QoS features Origin Server Precip (Primary) Origin Server Precip (Backup) wx-primary.noaa.gov:80 wx-backup1.noaa.gov:80 RegReps Top Tier Distribution Servers wx1.backbone.faa.gov:80 wx2.backbone.faa.gov:80 wx3.backbone.faa.gov:80 6: Service Request 1: Find services for SAS 'Precip' datasets Edge Tier Distribution Servers 4: Service Request 2: wx-primary.noaa.gov:80 wx-backup1.noaa.gov:80 • FTI support for latency includes VLANs and class-based fair queing, at router/switch level • Can use proxy service address resolver to provide appropriate network addresses to align with VLAN-based prioritization • Can also use resolver to route to class-based fair queues via the use of endpoints on same network, but with different port ranges (e.g., ports 10000-10999 are 'low priority' Critical Priority 5: Resolve proxy services High Priority 3: Resolve proxy services Med Priority Proxy service address resolver Low Priority

  15. Dynamic SAS Models • 'Slow' Dynamic SAS Model • Example: Weather avoidance field generated using new algorithm • Switchover can be handled via Registry infrastructure • Multiple days or weeks of warning • 'Fast' Dynamic SAS Model • Ensemble forecast with real-time performance validation • SAS data source (which forecast or blend of forcasts is best) can change on every update cycle for the product • Best thought of as a data fusion system, producing the appropriate SAS product • Unwieldy to think of Registry handling per-data-cycle SAS changes

  16. Backups

  17. Content Delivery Network Routing Design Goals • Make distribution infrastructure as transparent as possible to client • Keep routing logic relatively simple • FAA data access patterns for weather not highly dynamic • Similar to simple broker network concept for SWIM • Allow 'pure distribution' portion of infrastructure to function without a RegRep (non FAA SAS use cases) using static service endpoint • Also support caching of RegRep info in FAA SAS use case for fault tolerance • Align with current/envisioned QoS capabilities of FTI

  18. Hub & Spoke Topology Examples:Server Cluster + Proxies, WAN Weather Data Clients Weather Data Origin Server • Scales well for situation where there are many users at remote locations with overlapping data requests (common FAA use case) • Requires intelligent proxy servers • Most complex, but most capable, topology • Variants exist for fault-tolerant scenarios, but basically the same Router Weather Data Proxy Server w/Cache FTI WAN Weather Data Proxy Server w/Cache Router Weather Data Proxy Server w/Cache Weather Data Clients Weather Data Clients

  19. Federated Query for the 4-D Cube • Federated model allows for decentralized governance • Assumes that reliability/performance of partner organizations is ‘impedance matched’ • Simple Redundant registries within each domain NOAA Registry FAA Registry NWS Registry NOAA Registry FAA Registry NWS Registry Oceans Register AIM Register Other Register 4-D Cube Register 4-D Cube Register 4-D Cube Register • Query is ‘fanned out’ to all three registries • Duplicate results, if present, are pruned • Issues: Additional latency, infrastructurereliability mismatches, no FAA governancein the loop for partner organization changes FAA Registry Client (Human or machine)

  20. Main Server / Edge Server Notional OGC-Compliant Block Diagrams Main Server Edge Server FTI WAN Real-Time Data Sources Router Server Write Interfaces Server Read Interfaces Client Read Interfaces Server Read Interfaces DB Query Engine Intelligent Query Proxy DB Life-Cycle Manager WFS-T WCS-T WFS/WCS Req/ Reply WFS/WCS Req/ Reply WFS/WCS Req/ Reply Query Manager Query Request Combiner Query Response Splitter/ Filter File Watcher/ Scanner Pub/ Sub Pub/ Sub Persistent Query Manager WFS/WCS Pub/ Sub Ext NFS Direct IO Create/ Update Delete Data Store Disk Cache Mem Cache HTTP HTTP Server HTTP HTTP HTTP Proxy Server

  21. Proxy –Repeater Pattern

More Related