AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR

1. AWIPS Continuous Technology Refresh (CTR)AWIPS Software CTR TO8: Outbrief February 7, 2008

2. T08 Outbrief Introduction Intro • AWIPS-II T08 Summary • AWIPS-II Architecture Update • AWIPS-II CAVE Summary • AWIPS-II Workstation (CAVE) Capabilities • AWIPS-II SOA services • AWIPS-II Plug Ins • AWIPS-II Library • AWIPS-II Data • AWIPS-II TO8 Testing

3. Weather System Distributed User Needs:Drives Architecture Towards Advanced Solutions Summary Totally Non-Propriety Solution (Open Source) First Person Shooter Video Game Weather Edge User Visualization Quickly Adaptable to New Science & Data Types Accelerate Innovation & Speed of Deployment Google Earth Sat Com Increasing Data Interactions CAD Systems Situational Awareness Visualization Enabling of Con-Ops Flexibility: 164 Centers Increasing Data Rates High Data Rates With Gaming Style Interaction

4. TO8 Task Order Summary • Primary task is to migrate D2D workstation capability to the new AWIPS-II developed architecture • Performed workstation capabilities migration • Performed server side migration for ingest and storage of data • Perform risk reduction for GFE • Perform risk reduction for Hydro

5. AWIPS-II Central Architectural ConceptESB SOA Plug In Adaptable Future T08 Architecture Eclipse RCP Visualization Plug Ins SOA Data Type Plug Ins Airep Binlightening Bufrua Gfe Grib Obs Pirep Radar Recco Satellite Sfcobs Shef Taf Text AWIPS-II Services 14 Core 8 User Interface 22 Library 14 Application Staging Ingest Gfe VTEC Purging Index Archive Notify Adapter Utility AutoBld Product CAVE Shef Proxy /Light Thick /Light Client Mode Enterprise Service Bus

6. Mule ESB + SPRING ContainerServices decoupled from transport mechanisms Architecture MULE ESB SEDA Load Balanced Queue Mule End Point Provider Inbound Router Data Data Message Receiver Connector Transformers Mule Interceptor Pattern Logger Mule End Point Provider Outbound Router Profiler Message Receiver Security Check Transformers Connector EDEX SOA Service EDEX SOA Service • JMS • Files • HTTP • FTP • E-Mail • *** • EDEX SOA services are Dependency Injected Components • Configured in the XML Mule files Data Data

7. Plug Ins Pattern RefinementsDynamically Extends to New Data & Transforms Architecture Auto Generated HDF5 schema for Blob data ESB Endpoint Config Plug In (i.e)plugin-airep Metadata DB HDF5 Lighting Radar Satellite Grid GFE Auto Generates Schema Creates Data URI Specific Data Type Decoding Custom Data Transforms if necessary Custom Data Access Plug In Registry Table

8. Data Repository Maintenance RefinementBased on a circular cycle of tables & HDF5 files Architecture PurgeSrv 1 hr Wake Up DbManager Table holds retention period /2 of data (I.e. 12hr of data for 24 hr of retention time) HDF5 directory tree Plugin Metadata day & hour 30 1 delete Metadata_1 Metadata_2 2 Metadata_3 Delete Referenced HDF5 files new Metadata_4 Purge creates new table and deletes oldest

9. Clustered Data Ingest Flow Refinement:Pass References and MetaData between Services Architecture ../sbn/… Airep Binlightening Bufrua Gfe Grib Obs Pirep Radar Recco Satellite Sfcobs Shef Taf Text EDEX file reference StagingSrv singleton ../../processing clustered queue jms://cp/… read & delete IngestSrv f(plugin) data record HDF5 vm://… metadata persist IndexSrv f(plugin) RDBMS List records i.e. MetaData vm://… Alert Topic NotifySrv threaded Subscriptions

10. AutoBldSrv IngestSrv NotifySrv IndexSrv postGreSQL Table: Subscription Table: scripts CAVE Data Notification + SubscriptionData Matched to Menus by Optimized Decision Tree Architecture Data Item EDEX CAVE DecisionTree CAVE Startup Quinlan ID3 Algorithm AlertJobStarter Build Data URI Menu dataURIs URICatalog Message Received AlertManagerJob Data URI List quartz Timer ingested dataURIs Data URI Aggregator EDEX Script Ids topic:edex.alerts topic:edex.subscription Check scripts for URIs

11. Workstation CAVE Architecture OverviewBuilt from Eclipse RCP Plugins + Extensible CAVE-Summary Eclipse 3.3 XML User Customizable Menus & Tool bar GIS Display defined by map projection and set of layers (history recallable, procedures) High Performance Image Rendering by GPU Shader Language Controls of (Projection,Color, …) Large raster data set rendering through Quad Tree Tilling and HDF5 chunking Active Raster and Vector Data Interrogation Dynamic map re-projection using GeoTools Transforms& GPU Solid Shape Warping Auto updating Animations of all data types in all panes ESB Connection for thin/thick client access to data Enterprise Service Bus Data Connections

12. CAVE leverages Eclipse PerspectivesEnables Multiple User Interfaces CAVE-Summary Weather Workstation Hydrology Workstation Graphical Forecast Editing • Perspectives leverages entire set of CAVE eclipse plug ins • Each perspective can have unique menus, toolbars, and dialogs • User can actively switch back and forth between perspectives

13. Map Reprojection: Texture WarpingGPU Fragment Shader 50 times Faster than CPU CAVE-Summary • Map Reprojection at Interaction Speeds • Interactive Color Map HDF5 Repository Eclipse RCP Visualization Chunked Float Array Mapping Vertex Array HDF5 Library IMAGING_GLSL_PROGRAM Color Map 1D Texture RGB Color Map XML RGBA quads OpenGL API JOGL Control • Raw data sampling enabled by the use of raw data CPU-GPU Boundary Graphical Processing Unit (GPU) GPU Commands & Data Streams IMAGING_GLSL_PROGRAM

14. Workstation Menus / Toolbars / RenderingXML Configurable Menus + XML Bundles Workstation .ui.personalitles.awips plugin.xml .site. (from localization) plugin.xml XML Display Resource Bundles Commands ~19 Map plugin.xml Commands Parameters Menu labels ~11 Plots setScale ~30 Display Resources RadarDisplayControls 1 Skew-T menuRetrieval productURI productName variableList 8 Radar 11 Models

15. Workstation ConfigurationXML used for most all configuration Workstation CAVE configuration files under cave/etc bundles 121 RGBA list unlimited colormaps 17 svg, 12 spi plotmodels XML Menus volumebrowser Templates + config warngen radarInfo contourStyleRules imageryStyleRules graphStyleRules

16. Workstation ModesImplemented Using Eclipse Preferences Workstation VTEC Product class code Test phrases in Body of Warning

17. Workstation (CAVE) Localization PatternUses UtilitySrv with a HTTP interface Workstation <<SOA Service>> UtilitySrv ESB Sync Protocal CAVE Config At CAVE Start Up Log In User Name Determines User Context Site Through Perferences CAVE Menus Localization Hierarchy Color Maps Base Local Persistence Of Current Active Localization Site User

18. Workstation History and ProceduresD2D Style Con-Ops (except Alter Bundle) Workstation Bundle Components Map Scale Definition Zoom Level Center Point Coord. Map Projection Grid Range <XML> Bundle Load Mode Layer Definitions Metadata Fields / File Name • Menus Also Use • Bundles • Easy Customization of Menus • Bundles are reuseable through the passing in of arguments Color Map Display Controls Need to add XY Components

19. GIS Map Display Defined by XML BundleEnables reuse in D2D style Menus Workstation Map Scale Definition Zoom Level Center Point Coord. Map Projection Grid Range <XML> Bundle Load Mode Layer Definitions Metadata Fields / File Name Display Bundle is automatically built as data is selected Time matching control Color Map Display Controls Map Bundles Available Under Maps Menu Pull Down

20. Workstation Volume BrowserRetrieves Views of Data From Grid Repository Workstation TimeMatcher RequestJob Loader PlanViewDisplay HDF5 Grid Repository {…}Resource TimeMatcher Loader VarHeightDisplay Direct Port of D2D Time Matching SkewTDisplay {…}Resource TimeHeightDisplay Point | Line CrossSection…Display Data Cube Slice

21. Workstation Volume Browser MenusKOAX menu & Basic included in installation Workstation XML Menu Config cave/etc/volumebrowser KOAX Localization Basic User Manual ADE 1.0 Volume Browser Under CAVE menu Rename Files To Create Basic Version BrowserPlanViewPlanes -> BrowserTimePlanes BrowserSpaceFieldMenu -> BrowserSpaceFields, BrowserCrossSectionSpaceFields, BrowserCrossSectionTimeFields, BrowserHeightFields BrowserTimeFieldMenu -> BrowserTimeFields BrowserSourceConus -> BrowserAllSources

22. Workstation Plot Model and MaintenanceD2D Plot Vector Library Converted to SVG (XML) Workstation Metars Obs MetaData & Parsed data fields {…} Plug In Job PlotModel Generator Buoys PlotResource Progressive Disclosure SVG Plot Model SVG Plot Model SVG Plot Models Entire D2D present Wx symbol set converted to SVG UA plot 925MB

23. Workstation Color Map EditorReplicates D2D with transparency control Workstation Color Maps Stored on Server Eclipse RCP Plug In .viz.ui Also, set background color dialog UtilitySrv SaveColorMapDialog ../cave/etc/colormaps/.. Local Storage Initialized During Localization Color Map XML Format (unlimited number of colors) <colorMap> <r>..</r> <g>..</g> <b>..</b> <a>..</a> </colorMap> ../data/utility/cave_static/base/colormaps ./site/colormaps/koax ./user/colormaps

24. Workstation Text WS and EditorImplemented as 2 RCP plug ins in CAVE Workstation .texteditor .textworkstation EDEX TextDB Workstation Interacts with TextDB text Plug In Hibernate Access Objects {tablename}Dao ../opt/data/sbn/test Text Ingest Endpoint

25. Workstation Warning Generation T09+ Delivered T08 Delivered Workstation WarnGen GUI Life Cycle Rules Dissemination VTEC Database tornado.vm severethunderstorm.vm Templates 3..19 TextDB GIS Capabilities Warning by PolyGon UPDATE LIST Severe Weather Statement Store Set VTEC Generate Text Send to TextWS Text Workstation Editor Restart

26. Workstation Radar Display T09+ Delivered T08 Delivered Workstation Radar Plug In Radar HDF5 Storage Render Radial Graphic Decode Render Raster Radial Decode Render Graphic Text Decode Store Radar Radial Products Raster Decode Store Radar Raster Products Store Radar Graphic Products TextDB Post Radar Text Products

27. Workstation Skew-T DisplayDisplays both UA and Model data Workstation CAVE RCP Plug In .viz.skewT Meteo Library SkewTDisplay SkewTResource GribAdapter UAObsAdapter Datacube EDEX bufrua Plug In Meta DATA Dependency on Fortran function with problem in JAVA interface Grib Plug In HDF5

28. SOA ServicesUtility, Maps, Radar Message Handler SOA • SRTM Plus from UCSD • SRTM30 products from USGS EROS • Ocean data from LDEO project • 30sec resolution (~1km) Stored in decimated blobs to support Quad Titling ./CAVE/topo ./CAVE/basemaps • Map Overlays Imported: • BCD Files • GeoTIFFs • LPI Files • Shapefiles • SPI Files TopoQuery +getHeight() +getTopoPoint() +getTopoLine() +getTopoArea() METARs Used to support disclosure in Plot models.

29. T08 Radar Implementation Reuse of AWIPS I NEW AWIPS II SOA SBN NCF Level 2 Filtered Level 3 Products DXn Server ORPG Level 3 Products ORPGCommsMgr ESB Container (EDEX) Radar Plug In ../radar/* RDBMS MetaData HDF5

30. SOA Data Type Plug Ins Plug Ins binlighting.h5 • Bin lighting (plugin-binlightning) • GINI Satellite (plugin-satellite) • Grib 1 and 2 (plugin-grib) • BUFR with RAOB (plugin-bufrua) • Aircraft (plugin-airep, plugin-pirep, plugin-recco) • Synoptic (plugin-sfcobs) • Buoy, Metar, CMAN, Land, MAROB, Mobile, SHIPS • Radar (plugin-radar) • TAF (plugin-taf) • METAR (part of Synoptic) • Text (plugin-text) • Collectives and StdText satellite.h5 grib.h5 radar.h5

31. Library Capabilities Library • VTEC management • Meteo Library • Text Generation (XML) • Switched from ANTLR parsing of existing templates to creating new Velocity based templates • Output text remains the same • Generalized Time

32. VTEC Management as a ServiceInitial User is WarnGen Text Generation Library Vtec API New SOA Service http WarngenAction +updateVtec +runTemplate VtecFactory VtecSrv WarnGen GUI VtecEventRetrieval.js VtecEventUpdate.js VtecObjectRetrieval.js VtecRetreival.js Hibernate Access Objects {tablename}Dao VTEC Database Warning Templates tornado.vm,cfg Severethunderstorm.vm,cfg Text WorkStation Text Data Base

33. D2D Meteo Library Library Controller +add_array() ************** +ztopsa() T08 Component Users ContourSupport fortconfbuf() calcrh() Mixrat() uvcomp() GlueGen Java JNI Interface Code hgt2pres() TopographyGraph SkewtDisplay libmeteoLib.so 127 FORTRAN Modules 17 C Modules

34. Data Management Data • Data access refinement (4D data cube) • SPRING Data Base Connection Pooling (T06 ran with 1 connection) • DaoPool Data access object pooling for improved performance • Data repository refinement • Added automatic index creation on sub tables • Hydro IHFS repository + Object to Relational Mapping (Hibernate) • Text Data Base + Object to Relational Mapping Direct Port of AWIPS-I TextDB Data Access Objects Dao Classes Dao Classes Dao Classes

35. DaoPool Dao Classes Dao Classes Dao Classes Data Access Improved PerformanceBy managing data access objects Data Singleton Example Use Developed mid T08 to improve ingest performance where queries need to be performed before inserts Pirep Plug In PirepDecoder PIREP MetaData

36. Hibernate Data Access Objects Dao Classes Dao Classes Dao Classes Hydro IHFS OB8.1 Data base tables Data Middlegen used to auto Generate relational to object mapping files IHFS 180 Tables OB8.1 One for each table

37. T08 Omaha Test EnvironmentDevelopment and Test Hardware Resources Testing Omaha Raytheon Server Room CP Processor Dell 2550 Raytheon Firewall awipscm Dell Server Unidata LDM-6.6.5 LDM-6.6.5 Auto Nightly 2 Week Stability Testing awips-dev1 2950 Server awips-int2 2950 Server awips-int1 2950 Server Fiber Line LDM-6.6.5 LDM-6.6.5 filters filters EDEX Cluster EDEX Nightly EDEX BldXX EDEX BldXX DVB-S Orion Network awips-db 2950 Server awips-nas 2950 Server postgreSql Shared NFS

38. T08 Integration Test Bed Metrics Testing Cluster Balance

39. T08 AWIPS-II Cumulative Memory ProfilesSmall Leak Noted: Working with profiling tools Testing JMX cpu Useage 0.41% Day Average Example Daily Profile