OPeNDAP Hyrax Back-End Server (BES) Customization

1. OPeNDAP HyraxBack-End Server (BES)Customization ESIP Federation Workshop 15 July 2008 Patrick West James Gallagher

2. BES Commands OLFS BES Module Module Module Module DAP2 THREDDS HTML XML- encapsulated object Java Servlet Engine Unix Daemon Optional THREDDS catalogs File system with data files, SQL Database, … OPeNDAP’s Hyrax Architecture

3. In what ways can you extend the BES? • Extensions are dynamically loaded from a shared object module • Containers and Definitions, and ways of storing them • Dataset Catalogs • Interface • Initialization/Termination • New commands (like our hello world example) • New response objects • New response handlers • New request handlers (data handlers like hdf4, hdf5, …) • Aggregation engines • Methods of returning your data (return as netcdf) • Reporters • Exception Handlers • Debugging

4. Modules are dynamically loaded into the BES • Within the BES configuration you specify the modules that you want to load into the BES. • The module is derived from the C++ abstract class BESAbstractModule. • The class impelements a C-style function called maker that instantiates the module class • The class implements an initialization and termination method to set up and clean up the module. • The module can add named handlers to the different pieces of the BES. • The module can extend already existing modules, for example extending the DAP2 module. • The module can modify already existing modules, such as changing the way certain responses are built.

5. BES Configuration File/usr/local/etc/bes/bes.conf

6. Running what we have • Let’s see what we have already installed • besctl start (if it’s not already running) • bescmdln -h localhost -p 10002 • show help; • show version; • show keys; • set container in catalog values c,data/nc/fnoc1.nc; • define d as c; • get dds for d; • define d as c with c.constraint=“u”; • get dds for d; • get ascii for d;

7. Or use the OLFS • ${CATALINA_HOME}/bin/startup.sh • Start your browser • Localhost:8080/opendap/ • Select data, then nc, then fnoc1.nc • Click the check box next to the variable u • Click the button ‘Get ASCII’

8. "Station<String>","latitude<Float32>","longitude<Float32>","temperature_K<Float32>","Notes<String>”"Station<String>","latitude<Float32>","longitude<Float32>","temperature_K<Float32>","Notes<String>” "CMWM",-34.7,23.7,264.3, "BWWJ",-34.2,21.5,262.1,"Foo” "CWQK",-32.7,22.3,268.4, "CRLM",-33.8,22.1,270.2,"Blah” "FOOB",-32.9,23.4,269.69,"FOOBAR" The CSV Data Handler Module • /root/src/bes-3.6.2/csv-handler • …more ‘data/temperature.csv’

9. Setting up the example • cd /root/src/bes-3.6.2/csv-handler • autoreconf • ./configure • make install • This will build and install the module library and the sample data • make bes-config • This will make the changes to the bes.conf file

10. BES Configuration File/usr/local/etc/bes/bes.conf

11. Restart it all • ${CATALINA_HOME}/bin/shutdown.sh • besctl stop • besctl start • ${CATALINA_HOME}/bin/startup.sh • Back in the browser • localhost:8080/opendap/ • Select data, then csv, then temperature.csv • Click the button ‘Get ASCII’

12. CSVModule class • Example Module class: • see /root/src/bes-3.6.2/csv-handler/CSVModule.cc • Implements initialize and terminate methods (and dump) • Adds: (typical for data handlers) • new request handler • new container • new catalog

13. Containers • A container represents data • Typically a data file • Can represent a request to a remote system, such as WCS • Has three main parts: • Symbolic name (‘cool_data’) • Real name (e.g., a filename) • Type (nc, ff, csv, ...) • matches the name of the data/request handler • Derive from the abstraction BESContainer • The BESFileContainer is built into the BES • You might add ... WCSContainer

14. Accessing the container • Implement the access method: virtual string access() ; • Returns the data container to be accessed, typically full path to a file. • BESFileContainer returns the stored file name • A WCSContainer might make a WCS request to the remote server, store the resulting data file, and returns the path to that resulting, cached, data file.

15. Definitions • Analogous to a view of data in a RDB • Can encompass many containers (available soon) • Can specify constraints for each container • Can specify data attributes from each container • Specify aggregation engine and command define d as <c1>[,…,<cn>] [with <c1>.constraint=“…”,<c1>.attributes=“…”, <cn>.constraint=“…”,<cn>.attributes=“…” aggregate using <engine> by <command>];

16. Dataset Catalogs • OLFS uses BES and THREDDS catalogs for dataset catalog traversal • Can have multiple catalogs in a single BES representing any type of catalog (coming soon) • Filesystem • Database catalog • Each Catalog is configurable • Catalogs inherit from the abstraction BESCatalog • Add to the list of catalogs in the module class: BESCatalogList::TheCatalogList()->add_catalog( catalog ) ;

17. Typical Catalog Configuration • Data handlers, e.g. csv-handler, add a BESCatalogDirectory representing a file system. • In the BES configuration file: • BES.Catalog.catalog.RootDirectory - data root directory of the catalog named “catalog” • BES.Catalog.catalog.TypeMatch - data types matched by file extensions • BES.Catalog.catalog.Include - what nodes to include in the catalog list (usually everything) • BES.Catalog.catalog.Exclude - what nodes to exclude (e.g. dot files) • Let’s look at /usr/local/etc/bes/bes.conf

18. Catalogs and Containers • Can create containers within the context of catalogs using BESContainerStorageCatalog • Uses BES.Catalog.catalog.TypeMatch to determine data type. set container in catalog values c,<catalog_file>;

19. Request Handlers • Derived from BESRequestHandler • Fills in response objects • Register functions that say “I know how to fill in some response”

20. Registering a Request Handler • Inside the Module class the request handler is registered with the BES BESRequestHandlerList::TheList()->add_handler( name, handler ) • CSVModule::initialize() BESRequestHandlerList::TheList()-> add_handler( modname, new CSVRequestHandler( modname ) ) ;

21. I know how • Inside the constructor for the request handler you register functions that know how to fill in responses add_handler( “name”, function ) • CSVRequestHandler::CSVRequestHandler add_handler( DAS_RESPONSE, CSVRequestHandler::csv_build_das ) ; add_handler( DDS_RESPONSE, CSVRequestHandler::csv_build_dds ) ; add_handler( DATA_RESPONSE, CSVRequestHandler::csv_build_data ) ; add_handler( VERS_RESPONSE, CSVRequestHandler::csv_build_vers ) ; add_handler( HELP_RESPONSE, CSVRequestHandler::csv_build_help ) ;

22. Definitions def as c1, c2 get das for def Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ff show help Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func das def def das das csv csv csv c1 c1 c1 c2 c2 c2 das help help das csv help How does it work command

23. Putting it together so far

24. Summary: CSV Data Handler Module • Look at CSVModule • Adds CSVRequestHandler • Adds BESCatalogDirectory • Adds BESContainerStorageCatalog • Look at CSVRequestHandler • Adds functions to fill in DAS, DDS, DataDDS, Version and Help responses • These functions fill in the respective response objects

25. More information… • The Following slides provide more information on BES • Interfaces • Callbacks • Request plans • Commands • Response objects • Response handlers

26. The Interface • There is a single interface into the BES. • Inherits from the abstraction BESInterface • Installed version uses BESCmdInterface • CEDAR uses BESApacheInterface • Creating BESXMLInterface • Interface runs a set of steps: • initialize the BES environment • build the execution plan • execute the request plan building the response object • transmit the response object • log the status of the request • report on the request • end the request

27. BESDataHandlerInterface • BESDataHandlerInterface structure is carried throughout the execution of the commands • Holds on to the response handler • Stores the list of containers • Holds the action being taken • Holds map of name/value pairs to be used during execution • Holds an error object if an error/exception occurs

28. Init/End callbacks • Can register callbacks during the initialization and end steps static void BESInterface::add_init_callback( p_bes_init init ) ; static void BESInterface::add_end_callback( p_bes_end end ) typedef bool (*p_bes_init)( BESDataHandlerInterface &dhi ) ; typedef void (*p_bes_end)( BESDataHandlerInterface &dhi ) ; • Examples: • Authentication/authorization • Initialize database connections • Clean up database connections and files

29. Building the request plan • Derived classes of BESInterface implement: virtual void build_data_request_plan() ; • BESCmdInterface parses the incoming command string • BESApacheInterface translates information from apache and parses the incoming command string • BESXMLInterface parses the XML document • Creates a response handler to execute the command

30. Commands • String commands sent from client to server • Built in commands: • show help; (version, process, status, keys) • set container … • show containers; • delete container <container_name>; • delete containers; • define … • show definitions; • delete definition <def_name>; • delete definitions; • set context <name> to <value>; • show context;

31. DAP Commands • DAPCommandModule adds: • get das for <def_name> [return as type]; • get dds … • get ddx … • get dods … • show catalog [for “node”]; (info) • dap-server modules (www,usage,ascii) add: • get info_page … • get ascii … • get html_form … • Data handlers (nc, ff, csv) don’t add any new commands.

32. Response Objects • Derived from BESResponseObject. • BESDASResponseObject • BESInfo • No methods required to be implemented • Created by a BESResponseHandler • Filled in by BESResponseHandler or delegated to BESRequestHandler

33. Informational Response Objects • Built in informational response objects derived from BESInfo: • BESTextInfo • BESHTMLInfo • BESXMLInfo • BESSilentInfo • Each one formats an informational response according to its type • Can add new informational response objects. BESInfoList::TheList()->add_info_builder( name, function ) ; • Function instantiates the derived BESInfo object. • Created in BESResponseHandler instances by calling: BESInfo *info = BESInfoList::TheList()->build_info() ; • Set in BES configuration file: BES.Info.Type=<name> e.g. txt, html, xml

34. Response Handlers: • represent a specific response, such as a DAS response, a DDS response, a help response ... • know how to create the response object (DAS, DDS, BESInfo) • do not necessarily fill in the response object, but know how the response should be filled in • BESDASResponseHandler knows to create a DAS object and that it needs to go to each request handler for each container. • A version response handler knows to create an informational response object and that it needs to go to all registered request handlers. • A status response handler knows to create a text response object and fill it in with the status of the server. • know how to transmit the response object

35. Implementing a Response Handler • Inherits from the C++ abstract class BESResponseHandler (e.g. BESDASResponseHandler, BESHelpResponseHandler) • Implement the methods: virtual void execute( BESDataHandlerInterface &dhi ) ; • Creates the response object • Does the work to fill in the response object, or delegate to a request handler to fill in the response object virtual void transmit( BESTransmitter *transmitter, BESDataHandlerInterface &dhi ) ; • Transmits the resulting response object using the appropriate method on the transmitter (more later)