1 / 39

DAP Servers and Services

DAP Servers and Services. Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan Potter and Patrick West. Outline . Different DAP Servers Hyrax Architecture and customization Example Configuration of Hyrax.

thorntona
Download Presentation

DAP Servers and Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan Potter and Patrick West

  2. Outline • Different DAP Servers • Hyrax Architecture and customization • Example Configuration of Hyrax

  3. There are several different DAP servers • Hyrax, developed by OPeNDAP • TDS, developed by Unidata • GDS, developed by COLA • PyDAP, developed by Roberto De Almeida • Others we won’t cover: Dapper; FDS; CODAR; OPeNDAP Server3; et cetera • Terminology: We often say ‘server’ when describing what the technical press calls a ‘web application.’

  4. Hyrax • Supports multiple protocols • Data: DAP using HTTP/GET and HTTP/SOAP; Direct access (via HTTP); WCS/WFS funded, in development • Catalog: THREDDS; HTML directories • Data formats: In binary distribution: NetCDF; HDF4; HDF5; FreeForm; many more available as source code. • Includes ASCII data dump, HTML data access form, Info metadata page

  5. OLFS BES Data Hyrax Architecture • Two (or more) cooperating processes: • Front-end provides DAP interface • Back-end(s) read(s) data • Both parts can be customized • Front-end: different network protocols • Back-end: different data formats/systems • N-Tier design is flexible, secure

  6. TDS • Java Servlet network interface • Supports multiple protocols • Data: DAP; WCS; NetCDF Subset; Direct access (via HTTP) • Catalog: THREDDS • Data formats: NetCDF; HDF5; GRIB-1,2; NEXRAD; DORADE; BUFR; DMSP; GINI; more in development • Can also read from any other DAP server • Can serve aggregations

  7. HTTP Tomcat Server catalog.xml Application THREDDS Server • WCS • OPeNDAP • HTTPServer • NetcdfSubset NetCDF-Java library configCatalog.xml Datasets IDD Data motherlode.ucar.edu THREDDS Data Server

  8. PyDAP • Python native implementation from the DAP2 specification; WSGI application • Server uses run-time loaded modules • Protocols: • Data: DAP (HTTP/GET); WMS/KML; EditGrid/Google Spreadsheets; JSON • Catalog: THREDDS using a script/batch system; full support planned • Formats: CSV; netCDF; SQL Matlab; GRIB; HDF5

  9. PyDAP Server Architecture:Plugins and Responses

  10. GDS • Java Servlet network interface and C/C++ back-end • Supports multiple protocols: • Data: DAP (HTTP/GET) • Catalogs: THREDDS • Formats: GRIB; HDF4; netCDF; BUFR; Binary; GrADS Station • Can also read from many DAP servers • GDS can perform sophisticated server-side operations and build interim data sets as a result. • Server-side operations use GrADScommand syntax.

  11. Server datasets in any format supported by GrADS performs manages sessions, supports extended request types for analysis translates dataset analysis, upload operations names Binary GrADS GRIB, HDF interface DODS server Java batch NetCDF code libraries servlet mode Result cache holds temporary data (uploaded, generated by a previous operation, or transferred directly from another server) for use in remote analysis GrADS Data Server

  12. DAP Servers Summary • Four servers, all developed by different groups, were described (Hyrax, TDS, PyDAP and GDS) • Each supports DAP and THREDDS • Each has unique features: • PyDAP: WMS, KML and python scripting • GDS: Extensive server-side operations • TDS: Aggregation, WCS • Hyrax: Customizable, SOAP interface, WCS/WFS support funded • These are more properly called ‘web applications.’

  13. Hyrax Architecture • Front-end (OLFS) customization • Authentication & Authorization • Back-end (BES) customization • Single/Multiple Machine Installations • Security

  14. BES Commands DAP2 OLFS BES THREDDS HTML XML- encapsulated object Java Servlet Engine Unix Daemon Optional THREDDS catalogs File system with data files, SQL Database, … Hyrax Architecture

  15. The OLFS uses an ordered list of Dispatch Handlers. • Each handler on the list is asked if it can handle the incoming request. • The first handler on the list that claims the request gets to handle it. • List order is important, as some handlers (for example THREDDS) may claim to handle requests that should be handled by a different handler. • Handlers are free to do anything they need to handle a request: Contact a remote system/process, read files, spawn threads, et cetera. OLFS Dispatch Handlers

  16. Extending the OLFS • Extension ‘modules’ written in Java • Added to a directory within Tomcat • The new modules have complete access to the request information • Both HTTP GET and POST requests • There is some significant processing done before the handler is called: • Conditional GET Requests • Authorization & Authentication

  17. OLFS and Authorization/Authentication • Authorization & Authentication (Az/Ac) actually provided by Apache or Tomcat • Apache/Tomcat provides a range of Az/Ac features: • Realms: Role-based Az/Ac • Usernames & passwords -> Authorization • Roles -> Authentication • Realm info in RDBs, XML docs, LDAP, … • TLS/SSL: • Server and client Az/Ac • Secure data transmission • Tomcat provides single sign-on capabilities • Clients must support cookies • Suitable for portals

  18. BES BES Framework Network Protocol and Process start/stop activities PPT* Initialization/ Termination DAP2 Access Data Catalogs BES Commands/ XML Documents Commands** NetCDF3 HDF4 FreeForm Data Store Interfaces … *PPT: point to point protocol **Some commands are built in, most load at run-time

  19. BES Extensibility In what ways can you extend the BES? • New request handlers (data handlers like netcdf, hdf4) • New response handlers (Cedar's Flat, Tab, Info responses) • New commands (like our hello world example) • Initialization/Termination • Exception Handlers • Reporters • Container and Definition storage • Aggregation engines • Methods of returning your data (return as netcdf)

  20. Possible Installations • Both OLFS and BES run on one machine • OLFS on one machine and BES on another • One OLFS and several instances of BES on different machines • OLFS communicating with one or more BESs and other backend processors

  21. Installation Security • A separate issue from Ac/Az • The BES must be protected: • With a firewall or • TLS & Client certificates • Running the OLFS and BES on separate machines limits the scope of a compromise of the OLFS • Ensure that the BES, Tomcat and Apache all run with limited access to the server host

  22. Hyrax Architecture Summary • Hyrax can be installed on one machine or several • Installation security merits serious consideration • Authentication & Authorization are handled by the web servers (Tomcat and Apache)

  23. Hands on: Hyrax Configuration • Choices: • Single or Multiple machine • Single or Multiple back-end servers • Tomcat or Apache web server • Data formats • Catalog customization • Security • Testing • Command line tools for system administrators • Web browser

  24. APAC Workshop Configuration • Run a single BES and the OLFS on one host (the virtual machine running SLAX Linux) • Use Tomcat running on port 8080 • Data formats: NetCDF and FreeForm • No custom THREDDS catalog • No firewall

  25. APAC Hardware • SLAX Linux VMware Virtual Appliance • All the software needed has been built, installed and configured • Start the virtual machine now…

  26. Starting the Virtual Machine • Insert the CD-ROM • Drag the opendap_vm and Wintools folders to the Windows desktop • Copy the ISO image from the CD ROM to the harddisk (in a command window, mkdir C:\SLAX and then use copy or dd to copy the iso to the new directory). • The dd command is in the Wintools folder • The C:\SLAX directory is coded into the virtual machine; use Vmware Workstation/Fusion to edit • Open the opendap_vm folder and double click on the opendap_vm.vmx virtual machine configuration file or start VMware Player/Workstation/Fusion and open opendap_vm.vmx from there. • The virtual host will boot and the SLAX (slackware Linux with KDE) desktop will appear.

  27. Installing the Software • Already present on the virtual machine • But, if it wasn’t, you’d go to the OPeNDAP web site* and download: • Binaries or source for the BES and the data handlers needed. • Get the OLFS web archive file (which is a compiled java servlet). • You also need the Tomcat servlet engine * http://opendap.org/download/hyrax.html

  28. Location of Server Files on the Virtual Machine • On your virtual machine: • The Tomcat servlet engine is in /usr/local/javadev/apache/tomcat-5.5.12 (this is the value of $CATALINA_HOME) • The BES, data handlers and related source files are in /usr/local/src. The BES has been built and installed in /usr/local/ ($prefix) • The OLFS web archive file is $CATALINA_HOME/webapps/opendap.war • BES: bes.conf, found at $prefix/etc/bes/bes.conf • OLFS: olfs.xml and catalog.xml, found at $CATALINA_HOME/content/opendap

  29. Background: Starting the Server • Start the BES (back-end data processing component) • Use bescmdln to verify it’s working • Start Tomcat: This automatically starts all installed servlets • Servlets are installed by copying the .war file to the servlet’s webapps directory • Verify it’s working using a web browser

  30. Start the BES

  31. Verify the BES is running

  32. Start Tomcat & the OLFS • Typical steps: • Unpack the olfs jar-file • Copy the opendap.war file to Tomcat’s webapps directory • Start Tomcat • Since all but the last step has been done already, start Tomcat: • /usr/local/javadev/apache-tomcat-5.5.12/bin/startup.sh

  33. …terminal view

  34. Verify Tomcat is running http://localhost:8080

  35. …and Hyrax http://localhost:8080/opendap

  36. Complete the Configuration • Steps you would typically perform: • Security: Set up a firewall! Limit access to port 10002 to this host only • Custom catalogs: Edit the catalog.xml configuration file. By default automatic catalogs are generated • Logging: Edit the log4j.xml file. By default all accesses are logged

  37. Stopping Hyrax First, stop Tomcat using ‘shutdown.sh’ …then stop the BES using ‘besctl’

  38. Hyrax Configuration Summary • For our chosen configuration, we used only Tomcat, plus the Hyrax web application which consists of the BES and the OLFS • The tools bescmdln, getdap and a web browser were used to test the installation • There are a lot of options, but the default settings produce a working server • Security is a must for a web application; use a firewall to isolate the BES so only the local host can connect to it.

  39. Summary • There are a number of DAP-compliant servers (Hyrax, TDS, PyDAP and GDS) were described • All of the servers (read ‘web applications’) actually support several other protocols • While DAP does nothing to support cataloging data, THREDDS supports just that • Hyrax is customizable and can be installed in several different configurations

More Related