1 / 33

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR. TO6: Outbrief and RRD June 13, 2007. T06 Out Brief Topics Theme of T06: ADE Hardening and Refinement. T06 Out brief Introduction. Architecture Changes Enhancements Localization Pattern (CAVE and EDEX)

james-blankenship
Download Presentation

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AWIPS Continuous Technology Refresh (CTR)AWIPS Software CTR TO6: Outbrief and RRD June 13, 2007

  2. T06 Out Brief TopicsTheme of T06: ADE Hardening and Refinement T06 Out brief Introduction • Architecture Changes • Enhancements • Localization Pattern (CAVE and EDEX) • CAVE menu extensibility pattern • Plug In Extensibility • uEngine Refractor • Repository Refinement • Deployment Simplification • Server Install with GUI driven installer • Workstation Install • Hardening and Tuning • Server and Cluster Performance • Memory Leaks, Cluster State and locking, Dead Locks • JPROBE and JMX tools

  3. App Migration Architecture Migration Roadmap: ADE DevelopmentT06 Hardens the ADE, ready for Application Migration Architecture Changes 6/15/07 1/1/06 6/15/06 1/4/07 TO 6 TO 5 Raytheon IRAD TO 3 TO 4 • Tech. Reference Arch. • SOA service framework • Plug-in framework • Micro Engine • Core Base of Services • Core Patterns • Initial Open Source Project Integration • Release 0.0 • 1rst Delivered Code • CM / Deploy Pattern • SOA Templates • Data Plug-ins • Sub / notification • mEngine Scripting • CAVE • Release 0.1 • Regression Test Environ. • CAVE + • mEngine + • Remote Data Access (XML) • Dissemination • Release 0.2 • Server Execution Structure • Remote Admin • Legacy Adapters • Alerting Pattern • SOA Services + • CAVE++ • Release 1.0 • Localization Pattern • Server • CAVE • Stress Testing • Code hardening • Installation Simplification Add Server Execution All execution on workstation TO 7 Detailed Application Migration Planning

  4. ADE Lines of Code Count T03-T0630K written in the last year Architecture Changes • Includes numerous refactorings to improve code base T03 T04 T05 T06

  5. <<Java>> HDF5DataStore <<Java>> DataLayer <<abstract>> BaseDao SOA Architecture – Logical Layered ViewLayers Separated By Simple APIs Architecture Changes Client/Presentation Services JMX CAVE External Programs Mission Services Layer Mbean ProductSrv IngestSrv NotifySrv Hydro Models AdapterSrv AutoBldSrv PersistSrv IndexSrv SubscribeSrv UtilitySrv LAPS Security Services /Demilitarized Zone (DMZ) Enterprise Service Bus - Communication PurgeSrv StagingSrv FORTRAN/C/C++ Command Line Programs Data Access Layer T06 New Hibernate HDF5 API Localization Store Platform Layer PostgreSQL HDF5 Data Persistence Store Metadata Index Spatial Index

  6. SOA – Technical Reference ViewTechnical Reference Architecture Extends Architecture Changes Remote Management (JMX) Core Base of Services • Micro Engine • Plug-in Framework • Extensible XML model • Core SOA Services <Technical Reference Architecture> Services +Ingest Data +Request +Index Data +Store Data +Adapt +Auto Build Extend to a specific domain • Plug in specific libraries • Plug in data types, transforms +Subscribe +Collaborate <Technical Reference Architecture> SOA System Services (EDEX) Data Types Satellite Feed 4.5Mps Meta Data Transforms +domain libraries Data Scripts +mapping libraries Local Data <Visualization Framework> CAVE Service Interface To Data • Clean separation between data and visualization • Canonical XML data model • Scriptable Interface 2D/3D GIS Drawing Vis Plugin 1 Raster Render Collaboration Vis Plugin 2 Vector Render Vis Plugin n XY Render T06 New Auto Animate Localization

  7. AWIPS-II ADE High Level System Services SOA Services Running in an ESB Container Architecture Changes CORE SOA Services JMX Remote Service Management Client WAN Visible ADE 1.0 Updated T06 Mbean Mbean Mbean Mbean Mbean IngestSrv StagingSrv PersistSrv PurgeSrv IndexSrv ProductSrv AdapterSrv CAVE Visualization Client FileSystem Meta Data Index uEngine Exec Adapter Data Rendering METAR Plug In RDBMS via JDBC Spatial Data Base Manage Subscription JNI Adapter CAVE Bundles Radar Plug In HDF5 persistence GFE IFPServer Data Interrogation Satellite Plug In CAVE Procedures GRIB Plug In Wx Drawing Wx Warning Radar All Tilts Hydro Visualization Mbean Mbean Mbean UtilitySrv NotifySrv AutoBldSrv Collaboration uEngine Localization Data Subscription Notify XMPP Color Tables Maps and Topo Enterprise Service Bus – HTTP, JMS, Virtual Memory, File Endpoints Enterprise Service Bus – HTTP, JMS, Virtual Memory, File Endpoints Services Independent of End Points

  8. Localization PatternCentralizes Local Configuration … T06 Enhancements • Localization procedure occurs at startup • CAVE localization is controlled through Eclipse Preferences • Extended Eclipse Preferences to use a XML data store • Different approach to data since CAVE can work with large data sets less subsetting is required (ie Shape Files, topo) • Localization is provided through two simple, unified interfaces • Configuration for the server • Localization for the client, with server synchronization capability • Localization provides a multi-tiered configuration • Base (standard national configuration values), Site for Server (any values the site chooses to override) • Base, Site and User for Client (values that the user overrides)

  9. Localization PatternUses New SOA service UtilitySrv T06 Enhancements <<SOA Service>> UtilitySrv ESB Sync Protocal At CAVE Start Up <<Java>> UtilityManager +putFile() +getFile() +listFiles() Log In User Name Determines User Context Site Through Perferences User Site Base

  10. config.xml plugin.xml Localization PatternDefines the default screen, menus, configuration T06 Enhancements <<SOA Service>> UtilitySrv Configration Preferences Menus default-bundle.xml Default screen Optional uEngine Script

  11. Localization Data – Hierarchical Preferences, Menus, Data … T06 Enhancements Base: <configuration> <textureCardPreference>128</textureCardPreference> <textureMemoryPreference>384</textureMemoryPreference> <framesPerSecondPreference>25</framesPerSecondPreference> <tileBoundaries>false</tileBoundaries> <connectionMethod>jms</connectionMethod> <jmsServerAddress>tcp://localhost:61616</jmsServerAddress> <dataDirectory>/awips/opt/data/hdf5</dataDirectory> <fontMagnification>1.0</fontMagnification> </configuration> Site: <configuration> <siteName>KOAX</siteName> <siteFullName>Omaha</siteFullName> <siteType>WFO</siteType> <dataDirectory>/oax-awips/opt/data/hdf5</dataDirectory> </configuration> User: <configuration> <fontMagnification>1.25</fontMagnification> </configuration>

  12. CAVE Menu Extensibility – Eclipse 3.3Configuration Preferences, Menus, Data … T06 Enhancements plugin.xml Menus defined in plug in xml CAVE satellite plug In bundle.xml references bundle Menu fields update Automatically through the AlertManagerJob as data is ingested Embedded uEngine Script Retreives and transforms data for display

  13. CAVE Menu Extensibility PatternLeverages Capability in Eclipse 3.3 T06 Enhancements CAVE satellite Plug In plugin.xml Bundle IRWindow.xml uEngine Script to Retreive loop of data

  14. <<Eclipse Job>> MenuManagerJob +run() {Every 2 Sec.} CAVE Menu Extensibility PatternAuto updating menu data time indicators T06 Enhancements Perform Initial Catalog Queries To Populate CAVE Menus Build List of DataURIs from Menus CAVE Start Up Alerts JMS Topic IngestSrv

  15. Plug In RefinementT06 Reorganization of Eclipse Plug Ins T06 Enhancements JAVA JVM Eclipse RCP JAVA JVM SOA Service Container TAF SHEF RADAR SATELLITE GRIB METAR MESOWEST T06 Minor Refactor of Plug In Pattern Incremental Upgrades Enabled Through Plug Ins

  16. Plug In Enhancements CAVE refactored, SOA new uEngine Scripting T06 Enhancements • CAVE plug ins refactored in >30 separate plug ins following Eclipse best practices SOA Data type Plug Ins Plug In XML and Configuration Meta data definition files Plug In Implementation Classes uEngine Task Scripts Extends ScriptTask

  17. T06 CAVE Eclipse RCP Plug In TreeDependency tree T06 Enhancements viz edex core Core.gl radar grid ui Core.catalog Ui.glMap Ui.tools.gis warngen alerts Core.graphing Ui.tools.nav shapefile geotiff pngtile satellite volumebrowser drawing Ui.tools.map gfe Ui.tools.looping skewT hydro pointdata

  18. SOA Plug In Tool Added to ADEAdding using the tool to a training module T06 Enhancements • New with ADE 1.0: a Plug-in Creation tool. • The Plug-in Creation tool is implemented as an Eclipse plug-in. • Once installed, the Plug-in Creation tool may be used like any other element of the Eclipse IDE. • The Plug-in Creation tool automates most of the work involved in generating the initial files for a data-type plug-in. • Automatically creates the required directory structure for the plug-in. • Generates class stubs for the required Java classes. • Generates initial configuration and build files.

  19. uEngine Scripting Changed to JavaScript T06 Enhancements • Goal: Simplify task writing and have control structures in language • Uses Mozilla's freely available Rhino • Same JavaScript engine in JDK 1.6, except Mozilla version offers more features • Used in Eclipse • UengineScript class, simply call setScriptText(String javaScriptCode) and run() • Extend ScriptTask, Basic POJOs with an execute() method • No digester rules, No chain data • Tasks only operate on one object (loops at the script level)

  20. New uEngine tasks TermQuery – 84 AsciiToXml – 24 FileIn – 40 FileOut – 112 GribMap – 187 ColorMapImage – 78 ImageOut – 77 MakeResponseAscii – 30 MakeResponseUri - 45 Comparison of number of lines of code T06 Enhancements • Current uEngine tasks • TermQueryIndex – 297 • ASCIIToXML – 70 • FileIn – 106 • FileOut – 121 • GribMap – 215 • ColorMapImage – 110 • ImageOut – 114 • MakeResponseASCII – 37 • MakeResponseURI – 97

  21. New uEngine Script ExamplesSatellite and Grid uEngine examples T06 Enhancements Retreives Loops Sets Color Map Image Format

  22. T06 ProductSrv ChangesJavaScript uEngine Language & DB for Subscribe T06 Enhancements <<SOA Service>> ProductSrv uEngineScript(script) .run() Y Subscribe subscribe(subScript,dataURI,scriptID) N SubscribeDAO postgreSQL AWIPS-II script subscription End

  23. T06 NotifySrv ChangesScripts and Subscriptions are in RDBMS T06 Enhancements Architecture Summary <<SOA Service>> NotifySrv <<SOA Service>> AutoBldSrv Script Ids getDataURI() from New Ingested Data getModifiedScript (subscribeKey, scriptID, dataURI) Run Script getSubscriptionKey(dataURI) Build Response getSubscription () JMS Topic Subscription Return Output PostgreSQL Data Base Send list of script Ids that Depend on the Ingested Data subscription Compiled JavaScript uEngine script CAVE scripts

  24. T06 New service: UtilitySrvSupports Serving Out Localization Data T06 Enhancements <<SOA Service>> UtilitySrv Synro. Interface <<Java>> UtilityManager +putFile() +getFile() +listFiles() Localization Data T08 Color Tables Maps Topo

  25. Repository Refinement T06 Enhancements • Subscription added to PostgreSQL data base • Server state data simplifies clustering • Configurable Purge Retention Period • Performance enhancements with indexes on metadata PostgreSQL Data Base scripts subscription

  26. Spatial Table Refactor – Geo SpatialPostgis extension to PostgreSQL T06 Enhancements • Geo Spatial Enabling Data • Chosen Approach: Create Static Spatial Tables in PostgreSQL • PostGIS extension: Free, simple, high performance • Input / Output: • All ingested data is spatially indexed and can be spatially queried • Can create Shape and GeoTiff output PostgreSQL Data Base radar_spatial spatial_grids spatial_obs_stations spatial_satellite

  27. Unit Testing Framework Added T06 Enhancements ANT Build Control Files Cruise Control Automatic Build CM Code Repository Build.xml Test.xml <target test • Unit tests will be used to verify meteo lib functions • Unit tests will be used for automatic regression testing of common packages • Unit test can test lower level component parts of applications Junit test Source Files Supported by Eclipse IDE

  28. Deployment SimplificationBased on IzPack (Open Source Install Shield) Deployment • Created a new GUI based installer • Standard Linux Server Install for Server Side pieces • Standard Linux Workstation Install for CAVE • Windows Workstation for CAVE • Windows Workstation Server Side Using VMWare • Procedure for clustering (varies on Clustering Con-ops) • Linux Server 1 • Edex server • JDK • PostgreSQL • Eclipse 3.2 • ANT 1.7 • Edex source code • Edex Javadoc • Win Workstation • CAVE • Basemaps • JRE • Eclipse, ANT, src • Linux Workstation • CAVE • Basemaps • JRE • Win Workstation • CAVE • Basemaps • JRE • Vmware Linux Server • Edex server • JDK • PostgreSQL Remote Mount Data Partition

  29. Deployment Simplification Deployment • CAVE application and EDEX services installed using two separate installers • Each installer can install all files to the user’s home directory. • EDEX can be run as a VMWare image with CAVE interface Disk Space Check

  30. Hardening – JPROBE dynamic code analysis T06 Hardening Jprobe Execution Time Breakout For Grib Ingest Satellite Ingest 108ms 46ms/msg 64ms/msg • CAVE memory leak when running loops overnight, fixed • activeMQ dropping Mule connections (time outs), fixed • Jgroups configuration for clustering when clustering Con-ops determined • Message reference memory leak in AWIPS services fixed • Message file caching for idle messages eviction policy added to prevent memory growth

  31. Server Memory and CPU useageJMX Jconsole Cluster Monitoring T06 Hardening Dell 1850 2 Dual core/4G CP Processor Clustered Server awips-int1 Dell 1850 Clustered Server awips-int2

  32. ADE is Ready for ApplicationsRunning out of ideas for refactoring T06 Hardening • Core ADE patterns are stable and perform well • Visualization Performance is better than expected • CAVE uses Eclipse RCP 3.3 which has added XML menu extensions (a 250 slide SWT GUI guide has been put together): Ready for application interface coding • SBN data feed to Cluster has been active during the T06 and used to find memory and stability issues • SOA Plug In pattern decouples development (the IFPServer is being ported entirely within a plug in) • uEngine pattern is stable and turning out to be a very powerful enabler of adaptability • Other groups within Raytheon are showing increasing interest

  33. T06 RRD Scenario Steps Demo Scenario

More Related