350 likes | 482 Views
OPeNDAP Hyrax Back-End Server (BES) Customization. ESIP Federation Workshop 15 July 2008 Patrick West James Gallagher. BES Commands. OLFS. BES. Module. Module. Module. Module. DAP2. THREDDS. HTML. XML- encapsulated object. Java Servlet Engine. Unix Daemon. Optional THREDDS
E N D
OPeNDAP HyraxBack-End Server (BES)Customization ESIP Federation Workshop 15 July 2008 Patrick West James Gallagher
BES Commands OLFS BES Module Module Module Module DAP2 THREDDS HTML XML- encapsulated object Java Servlet Engine Unix Daemon Optional THREDDS catalogs File system with data files, SQL Database, … OPeNDAP’s Hyrax Architecture
In what ways can you extend the BES? • Extensions are dynamically loaded from a shared object module • Containers and Definitions, and ways of storing them • Dataset Catalogs • Interface • Initialization/Termination • New commands (like our hello world example) • New response objects • New response handlers • New request handlers (data handlers like hdf4, hdf5, …) • Aggregation engines • Methods of returning your data (return as netcdf) • Reporters • Exception Handlers • Debugging
Modules are dynamically loaded into the BES • Within the BES configuration you specify the modules that you want to load into the BES. • The module is derived from the C++ abstract class BESAbstractModule. • The class impelements a C-style function called maker that instantiates the module class • The class implements an initialization and termination method to set up and clean up the module. • The module can add named handlers to the different pieces of the BES. • The module can extend already existing modules, for example extending the DAP2 module. • The module can modify already existing modules, such as changing the way certain responses are built.
Running what we have • Let’s see what we have already installed • besctl start (if it’s not already running) • bescmdln -h localhost -p 10002 • show help; • show version; • show keys; • set container in catalog values c,data/nc/fnoc1.nc; • define d as c; • get dds for d; • define d as c with c.constraint=“u”; • get dds for d; • get ascii for d;
Or use the OLFS • ${CATALINA_HOME}/bin/startup.sh • Start your browser • Localhost:8080/opendap/ • Select data, then nc, then fnoc1.nc • Click the check box next to the variable u • Click the button ‘Get ASCII’
"Station<String>","latitude<Float32>","longitude<Float32>","temperature_K<Float32>","Notes<String>”"Station<String>","latitude<Float32>","longitude<Float32>","temperature_K<Float32>","Notes<String>” "CMWM",-34.7,23.7,264.3, "BWWJ",-34.2,21.5,262.1,"Foo” "CWQK",-32.7,22.3,268.4, "CRLM",-33.8,22.1,270.2,"Blah” "FOOB",-32.9,23.4,269.69,"FOOBAR" The CSV Data Handler Module • /root/src/bes-3.6.2/csv-handler • …more ‘data/temperature.csv’
Setting up the example • cd /root/src/bes-3.6.2/csv-handler • autoreconf • ./configure • make install • This will build and install the module library and the sample data • make bes-config • This will make the changes to the bes.conf file
Restart it all • ${CATALINA_HOME}/bin/shutdown.sh • besctl stop • besctl start • ${CATALINA_HOME}/bin/startup.sh • Back in the browser • localhost:8080/opendap/ • Select data, then csv, then temperature.csv • Click the button ‘Get ASCII’
CSVModule class • Example Module class: • see /root/src/bes-3.6.2/csv-handler/CSVModule.cc • Implements initialize and terminate methods (and dump) • Adds: (typical for data handlers) • new request handler • new container • new catalog
Containers • A container represents data • Typically a data file • Can represent a request to a remote system, such as WCS • Has three main parts: • Symbolic name (‘cool_data’) • Real name (e.g., a filename) • Type (nc, ff, csv, ...) • matches the name of the data/request handler • Derive from the abstraction BESContainer • The BESFileContainer is built into the BES • You might add ... WCSContainer
Accessing the container • Implement the access method: virtual string access() ; • Returns the data container to be accessed, typically full path to a file. • BESFileContainer returns the stored file name • A WCSContainer might make a WCS request to the remote server, store the resulting data file, and returns the path to that resulting, cached, data file.
Definitions • Analogous to a view of data in a RDB • Can encompass many containers (available soon) • Can specify constraints for each container • Can specify data attributes from each container • Specify aggregation engine and command define d as <c1>[,…,<cn>] [with <c1>.constraint=“…”,<c1>.attributes=“…”, <cn>.constraint=“…”,<cn>.attributes=“…” aggregate using <engine> by <command>];
Dataset Catalogs • OLFS uses BES and THREDDS catalogs for dataset catalog traversal • Can have multiple catalogs in a single BES representing any type of catalog (coming soon) • Filesystem • Database catalog • Each Catalog is configurable • Catalogs inherit from the abstraction BESCatalog • Add to the list of catalogs in the module class: BESCatalogList::TheCatalogList()->add_catalog( catalog ) ;
Typical Catalog Configuration • Data handlers, e.g. csv-handler, add a BESCatalogDirectory representing a file system. • In the BES configuration file: • BES.Catalog.catalog.RootDirectory - data root directory of the catalog named “catalog” • BES.Catalog.catalog.TypeMatch - data types matched by file extensions • BES.Catalog.catalog.Include - what nodes to include in the catalog list (usually everything) • BES.Catalog.catalog.Exclude - what nodes to exclude (e.g. dot files) • Let’s look at /usr/local/etc/bes/bes.conf
Catalogs and Containers • Can create containers within the context of catalogs using BESContainerStorageCatalog • Uses BES.Catalog.catalog.TypeMatch to determine data type. set container in catalog values c,<catalog_file>;
Request Handlers • Derived from BESRequestHandler • Fills in response objects • Register functions that say “I know how to fill in some response”
Registering a Request Handler • Inside the Module class the request handler is registered with the BES BESRequestHandlerList::TheList()->add_handler( name, handler ) • CSVModule::initialize() BESRequestHandlerList::TheList()-> add_handler( modname, new CSVRequestHandler( modname ) ) ;
I know how • Inside the constructor for the request handler you register functions that know how to fill in responses add_handler( “name”, function ) • CSVRequestHandler::CSVRequestHandler add_handler( DAS_RESPONSE, CSVRequestHandler::csv_build_das ) ; add_handler( DDS_RESPONSE, CSVRequestHandler::csv_build_dds ) ; add_handler( DATA_RESPONSE, CSVRequestHandler::csv_build_data ) ; add_handler( VERS_RESPONSE, CSVRequestHandler::csv_build_vers ) ; add_handler( HELP_RESPONSE, CSVRequestHandler::csv_build_help ) ;
Definitions def as c1, c2 get das for def Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ff show help Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func das def def das das csv csv csv c1 c1 c1 c2 c2 c2 das help help das csv help How does it work command
Summary: CSV Data Handler Module • Look at CSVModule • Adds CSVRequestHandler • Adds BESCatalogDirectory • Adds BESContainerStorageCatalog • Look at CSVRequestHandler • Adds functions to fill in DAS, DDS, DataDDS, Version and Help responses • These functions fill in the respective response objects
More information… • The Following slides provide more information on BES • Interfaces • Callbacks • Request plans • Commands • Response objects • Response handlers
The Interface • There is a single interface into the BES. • Inherits from the abstraction BESInterface • Installed version uses BESCmdInterface • CEDAR uses BESApacheInterface • Creating BESXMLInterface • Interface runs a set of steps: • initialize the BES environment • build the execution plan • execute the request plan building the response object • transmit the response object • log the status of the request • report on the request • end the request
BESDataHandlerInterface • BESDataHandlerInterface structure is carried throughout the execution of the commands • Holds on to the response handler • Stores the list of containers • Holds the action being taken • Holds map of name/value pairs to be used during execution • Holds an error object if an error/exception occurs
Init/End callbacks • Can register callbacks during the initialization and end steps static void BESInterface::add_init_callback( p_bes_init init ) ; static void BESInterface::add_end_callback( p_bes_end end ) typedef bool (*p_bes_init)( BESDataHandlerInterface &dhi ) ; typedef void (*p_bes_end)( BESDataHandlerInterface &dhi ) ; • Examples: • Authentication/authorization • Initialize database connections • Clean up database connections and files
Building the request plan • Derived classes of BESInterface implement: virtual void build_data_request_plan() ; • BESCmdInterface parses the incoming command string • BESApacheInterface translates information from apache and parses the incoming command string • BESXMLInterface parses the XML document • Creates a response handler to execute the command
Commands • String commands sent from client to server • Built in commands: • show help; (version, process, status, keys) • set container … • show containers; • delete container <container_name>; • delete containers; • define … • show definitions; • delete definition <def_name>; • delete definitions; • set context <name> to <value>; • show context;
DAP Commands • DAPCommandModule adds: • get das for <def_name> [return as type]; • get dds … • get ddx … • get dods … • show catalog [for “node”]; (info) • dap-server modules (www,usage,ascii) add: • get info_page … • get ascii … • get html_form … • Data handlers (nc, ff, csv) don’t add any new commands.
Response Objects • Derived from BESResponseObject. • BESDASResponseObject • BESInfo • No methods required to be implemented • Created by a BESResponseHandler • Filled in by BESResponseHandler or delegated to BESRequestHandler
Informational Response Objects • Built in informational response objects derived from BESInfo: • BESTextInfo • BESHTMLInfo • BESXMLInfo • BESSilentInfo • Each one formats an informational response according to its type • Can add new informational response objects. BESInfoList::TheList()->add_info_builder( name, function ) ; • Function instantiates the derived BESInfo object. • Created in BESResponseHandler instances by calling: BESInfo *info = BESInfoList::TheList()->build_info() ; • Set in BES configuration file: BES.Info.Type=<name> e.g. txt, html, xml
Response Handlers: • represent a specific response, such as a DAS response, a DDS response, a help response ... • know how to create the response object (DAS, DDS, BESInfo) • do not necessarily fill in the response object, but know how the response should be filled in • BESDASResponseHandler knows to create a DAS object and that it needs to go to each request handler for each container. • A version response handler knows to create an informational response object and that it needs to go to all registered request handlers. • A status response handler knows to create a text response object and fill it in with the status of the server. • know how to transmit the response object
Implementing a Response Handler • Inherits from the C++ abstract class BESResponseHandler (e.g. BESDASResponseHandler, BESHelpResponseHandler) • Implement the methods: virtual void execute( BESDataHandlerInterface &dhi ) ; • Creates the response object • Does the work to fill in the response object, or delegate to a request handler to fill in the response object virtual void transmit( BESTransmitter *transmitter, BESDataHandlerInterface &dhi ) ; • Transmits the resulting response object using the appropriate method on the transmitter (more later)