OPeNDAP Hyrax Back-End Server (BES)

1. OPeNDAP HyraxBack-End Server (BES) Original version presented at APAC 2007 OPeNDAP Workshop Patrick West

2. BES Commands Module DAP2 OLFS BES THREDDS Module HTML XML- encapsulated object Module Java Servlet Engine Unix Daemon Module Optional THREDDS catalogs File system with data files, SQL Database, … OPeNDAP’s Hyrax Architecture

3. In what ways can you extend the BES? • Extensions are dynamically loaded from a shared object module • Containers and Definitions, and ways of storing them • Dataset Catalogs • Interface • Initialization/Termination • New commands (like our hello world example) • New response objects • New response handlers • New request handlers (data handlers like netcdf, freeform, csv) • Aggregation engines • Methods of returning your data (return as netcdf) • Reporters • Exception Handlers • Debugging

4. Modules are dynamically loaded into the BES • Within the BES configuration you specify the modules that you want to load into the BES. • The module is derived from the C++ abstract class BESAbstractModule. • The class impelements a c-style function called maker that instantiates the module class • The class implements an initialization and termination method to set up and clean up the module. • The module can add named handlers to the different pieces of the BES. • The module can extend already existing modules, for example extending the DAP2 module. • The module can modify already existing modules, such as changing the way certain responses are built.

5. BES Configuration File/usr/local/etc/bes/bes.conf

6. Running what we have • Let’s see what we have already installed • besctl start (if it’s not already running) • bescmdln [-h localhost -p 10022] • show help; • show version; • show keys; • set container in catalog values c, /data/nc/fnoc1.nc; • define d as c; • get dds for d; • define d as c with c.constraint=“u”; • get dds for d; • get ascii for d;

7. Or use the OLFS • ${CATALINA_HOME}/bin/startup.sh • Start your browser • Localhost:8080/opendap/ • Select data, then nc, then fnoc1.nc • Click the check box next to the variable u • Click the button ‘Get ASCII’

8. The CSV Data Handler Module $HOME/src/hyrax-1.9/src/modules/csv_handler "Station<String>","latitude<Float32>","longitude<Float32>","temperature_K<Float32>","Notes<String>” "CMWM",-34.7,23.7,264.3, "BWWJ",-34.2,21.5,262.1,"Foo” "CWQK",-32.7,22.3,268.4, "CRLM",-33.8,22.1,270.2,"Blah” "FOOB",-32.9,23.4,269.69,"FOOBAR"

9. Setting up the example • cd $HOME/src/hyrax-1.9/src/modules/csv_handler • Autoreconf -fiv • ./configure • make; make check; make install • This will build and install the module library and the sample data • But, remember on the VM, this has all been done already…

10. BES Configuration File/usr/local/etc/bes/bes.conf

11. Restart it all • [${CATALINA_HOME}/bin/shutdown.sh] • besctl stop • besctl start • [${CATALINA_HOME}/bin/startup.sh] • Back in the browser • localhost:8080/opendap/ • Select data, then csv, then temperature.csv • Click the button ‘Get ASCII’

12. CSVModule class • Example Module class • $HOME/src/hyrax-1.9/src/modules/csv_handler • Implements initialize and terminate methods (and dump) • Adds: (typical for data handlers) • new request handler • new catalog • new container storage using the catalog

13. Containers • A container represents data • Typically a data file • Can represent a request to a remote system, such as WCS • Has three main parts: • Symbolic name (cool_data) • Real name - e.g. filename • Data Type (nc, h4, h5, ff, csv) - matches the name of the data/request handler • Derive from the abstraction BESContainer • BESFileContainer built in • WCSContainer

14. Accessing the container • Implement the access method: virtual string access() ; • Returns the data container to be accessed, typically full path to a file. • BESFileContainer returns the stored file name • WCSContainer makes the WCS request to the remote server, stores the resulting data file, and returns the path to that resulting, cached, data file.

15. Storing Containers • The default storage of containers is volatile for the duration of the session • Can create new persistent stores: • Store in database per user or group or project • Create class inherited from BESContainerStorage • Give the new class a unique name (user_store) • In the Module class add this new storage class to the container storage list BESContainerStorageList::TheList()->add_persistence( storage ) ; • Create containers in that storage set container in user_store values s,r,t;

16. Storing a container virtual void add_container( const string &sym_name, const string &real_name, const string &type ) ; • BESContainerStorageCatalog uses the catalog to create the full path to the file and uses the file extension to determine the data type. • WCSContainerStorage creates a WCSContainer whose access method makes the WCS request on the remote machine, caches the resulting file, and returns that as the file to access. • BESContainerStorageVolatile simply stores the container in a volatile stl map for the duration of the session.

17. Definitions • Like a view of data • Can encompass many containers (available soon) • Can specify constraints for each container • Can specify data attributes from each container • Specify aggregation engine and command define d as <c1>[,…,<cn>] [with <c1>.constraint=“…”,<c1>.attributes=“…”, <cn>.constraint=“…”,<cn>.attributes=“…” aggregate using <engine> by <command>];

18. Storing Definitions • Default storage is volatile for the duration of the session • Can create new persistent stores: • Store in database per user or group or project • Create class inherited from BESDefinitionStorage • Give the new class a unique name (user_store) • In the Module class add this new storage to the definition storage list DefinitionStorageList::TheList()->add_persistence( storage ) ; Create definitions in that storage define d in user_store as …

19. Storing a Definition virtual void add_definition( const string &def_name, BESDefine *def ) ; • BESDefinitionStorageVolatile simply stores the definition in a volatile stl map for the duration of the session.

20. Dataset Catalogs • OLFS uses BES and THREDDS catalogs for dataset catalog traversal • Can have multiple catalogs in a single BES representing any type of catalog (coming soon) • Filesystem • Database catalog • Each Catalog is configurable • Catalogs inherit from the abstraction BESCatalog • Add to the list of catalogs in the module class: BESCatalogList::TheCatalogList()->add_catalog( catalog ) ;

21. Typical Catalog Configuration • Data handlers, e.g. nc_module, add a BESCatalogDirectory representing a file system. • In the BES configuration file: • BES.Catalog.catalog.RootDirectory - data root directory of the catalog named “catalog” • BES.Catalog.catalog.TypeMatch - data types matched by file extensions • BES.Catalog.catalog.Include - what nodes to include in the catalog list (usually everything) • BES.Catalog.catalog.Exclude - what nodes to exclude (e.g. dot files) • Let’s look at $prefix/etc/bes/bes.conf

22. Catalogs and Containers • Can create containers within the context of catalogs using BESContainerStorageCatalog • Uses BES.Catalog.catalog.TypeMatch to determine data type. set container in catalog values c,<catalog_file>;

23. Putting it together so far

24. The Interface • There is a single interface into the BES. • Inherits from the abstraction BESInterface • Installed version uses BESCmdInterface • CEDAR uses BESApacheInterface • Creating BESXMLInterface • Interface runs a set of steps: • initialize the BES environment • build the execution plan • execute the request plan building the response object • transmit the response object • log the status of the request • report on the request • end the request

25. BESDataHandlerInterface • BESDataHandlerInterface structure is carried throughout the execution of the commands • Holds on to the response handler • Stores the list of containers • Holds the action being taken • Holds map of name/value pairs to be used during execution • Holds an error object if an error/exception occurs

26. Initialize/Termination callbacks • Can register callbacks during the initialization and end steps static void BESInterface::add_init_callback( p_bes_initinit ) ; static void BESInterface::add_end_callback( p_bes_end end ) ; typedefbool (*p_bes_init)( BESDataHandlerInterface &dhi ) ; typedef void (*p_bes_end)( BESDataHandlerInterface &dhi ) ; • Examples: • Authentication/authorization • Initialize database connections • Clean up database connections and files

27. Building the request plan • Derived classes of BESInterface implement: virtual void build_data_request_plan() ; • BESCmdInterface parses the incoming command string • BESApacheInterface translates information from apache and parses the incoming command string • BESXMLInterface parses the XML document • Sets up a response handler to execute the command

28. Commands • String commands sent from client to server • Built in commands: • show help; (version, process, status, keys) • set container … • show containers; • delete container <container_name>; • delete containers; • define … • show definitions; • delete definition <def_name>; • delete definitions; • set context <name> to <value>; • show context;

29. DAP Commands • DAPCommandModule adds: • get das for <def_name> [return as type]; • get dds … • get ddx … • get dods … • show catalog [for “node”]; (info) • dap-server modules (www,usage,ascii) add: • get info_page … • get ascii … • get html_form … • Data handlers (nc, ff, csv) don’t add any new commands.

30. Response Objects • Derived from BESResponseObject. • BESDASResponseObject • BESInfo • No methods required to be implemented • Created by a BESResponseHandler • Filled in by BESResponseHandler or delegated to BESRequestHandler

31. Informational Response Objects • Built in informational response objects derived from BESInfo: • BESTextInfo • BESHTMLInfo • BESXMLInfo • BESSilentInfo • Each one formats an informational response according to its type • Can add new informational response objects. BESInfoList::TheList()->add_info_builder( name, function ) ; • Function instantiates the derived BESInfo object. • Created in BESResponseHandler instances by calling: BESInfo *info = BESInfoList::TheList()->build_info() ; • Set in BES configuration file: BES.Info.Type=<name> e.g. txt, html, xml

32. Response Handlers: • represent a specific response, such as a DAS response, a DDS response, a help response ... • know how to create the response object (DAS, DDS, BESInfo) • do not necessarily fill in the response object, but know how the response should be filled in • BESDASResponseHandler knows to create a DAS object and that it needs to go to each request handler for each container. • A version response handler knows to create an informational response object and that it needs to go to all registered request handlers. • A status response handler knows to create a text response object and fill it in with the status of the server. • know how to transmit the response object

33. Implementing a Response Handler • Inherits from the C++ abstract class BESResponseHandler (e.g. BESDASResponseHandler, BESHelpResponseHandler) • Implement the methods: virtual void execute( BESDataHandlerInterface &dhi ) ; • Creates the response object • Does the work to fill in the response object, or delegate to a request handler to fill in the response object virtual void transmit( BESTransmitter *transmitter, BESDataHandlerInterface &dhi ) ; • Transmits the resulting response object using the appropriate method on the transmitter (more later)

34. Request Handlers • Derived from BESRequestHandler • Fills in response objects • Register functions that say “I know how to fill in some response”

35. Registering a Request Handler • Inside the Module class the request handler is registered with the BES BESRequestHandlerList::TheList()->add_handler( name, handler ) • CSVModule::initialize() BESRequestHandlerList::TheList()-> add_handler( modname, new CSVRequestHandler( modname ) ) ;

36. I know how • Inside the constructor for the request handler you register functions that know how to fill in responses add_handler( “name”, function ) • CSVRequestHandler::CSVRequestHandler add_handler( DAS_RESPONSE, CSVRequestHandler::csv_build_das ) ; add_handler( DDS_RESPONSE, CSVRequestHandler::csv_build_dds ) ; add_handler( DATA_RESPONSE, CSVRequestHandler::csv_build_data ) ; add_handler( VERS_RESPONSE, CSVRequestHandler::csv_build_vers ) ; add_handler( HELP_RESPONSE, CSVRequestHandler::csv_build_help ) ;

37. c1 csv c2 csv das def def c1 c1 c2 c2 das das das csv csv das How does it work Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ff command get das for def Definitions def as c1, c2 Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func

38. The CSV Data Handler Module • $HOME/src/hyrax-1.9/src/modules/csv_handler • Look at CSVModule • Adds CSVRequestHandler • Adds BESCatalogDirectory • Adds BESContainerStorageCatalog • Look at CSVRequestHandler • Adds functions to fill in DAS, DDS, DataDDS, Version and Help responses • These functions fill in the respective response objects

39. Transmit • Sends the response object back to the client • Two built in transmitters • BESBasicTransmitter • BESBasicHTTPTransmitter (adds header info) • These implement methods to transmit informational responses (BESInfo) • BESDapTransmit implements static methods to send DAS, DDS, and DataDDS responses • Adds named static functions to the two built in transmitters • BESResponseHandler instances know what functions to call on the transmitter to transmit their response objects

40. Adding Transmit Functions • BESDapTransmit BESTransmitter *t = BESReturnManager::TheManager()-> find_transmitter( BASIC_TRANSMITTER ) ; if( t ) { t->add_method( DAS_TRANSMITTER, BESDapTransmit::send_basic_das ) ; t->add_method( DDS_TRANSMITTER, BESDapTransmit::send_basic_dds ) ; t->add_method( DDX_TRANSMITTER, BESDapTransmit::send_basic_ddx ) ; t->add_method( DATA_TRANSMITTER, BESDapTransmit::send_basic_data ) ; }

41. Adding Transmit Functions BESTransmitter *t = BESReturnManager::TheManager()-> find_transmitter( HTTP_TRANSMITTER ) ; if( t ) { t->add_method( DAS_TRANSMITTER, BESDapTransmit::send_http_das ) ; t->add_method( DDS_TRANSMITTER, BESDapTransmit::send_http_dds ) ; t->add_method( DDX_TRANSMITTER, BESDapTransmit::send_http_ddx ) ; t->add_method( DATA_TRANSMITTER, BESDapTransmit::send_http_data ) ; }

42. Adding new Transmitter • Inherit from BESTransmitter • Implement send_text, send_html • Add static functions to transmit other response objects in the constructor • E.G. ESG project, transmitter to return data object as a netcdf file get dods for <def> return as netcdf;

43. Aggregation • You can have multiple aggregation engines installed in the BES • Inherit from BESAggregationServer • Implement aggregate method: virtual void aggregate( BESDataHandlerInterface &dhi ) ; • In Module class add aggregation function that instantiates your instance BESAggFactory::TheFactory()->add_handler( name, function ) ; typedef BESAggregationServer *(*p_agg_handler)(string name);

44. Reporters • After command has been executed and response object transmitted • Inherit from BESReporter • Implement method: virtual void report( const BESDataHandlerInterface &dhi ) ; • Register with reporter list: BESReporterList::add_reporter( name, reporter_instance ) ;

45. Exception Handling • Register exception handler functions with the Exception Manager virtual void add_ehm_callback( p_bes_ehm ehm ) ; typedef int (*p_bes_ehm)( BESException &e, BESDataHandlerInterface &dhi ) ; BESExceptionManager::TheEHM->add_ehm_callback( func ) ; • When exception caught, passed to Exception Manager • Iterates through registered functions • If not handled, default handler • Set error_info in BESDataHandlerInterface • This is transmitted in place of response object

46. Logging and Debugging Informational and Debug Logging in the BES The BES has two forms of logging. • Event logging in a bes.log file. This is configured in the bes.conf file: • BES.LogName=/path/to/bes.log • BES.LogVerbose=no • Debug logging for the BES • besctl start -d “/path/to/bes.debug,nc,bes,ppt”

47. BES Debugging • Format of the command line option -d • “cerr|<log_file_name>,<context1>,…,<contextn>” • E.g. -d “/tmp/bes.debug,nc,bes” • Specify a context of “all” to turn on debugging for everything • Writing debug statements in your code • Macro BESDEBUG( “<context>”, <output to ostream> ) • E.g. BESDEBUG( “nc”, “reading DAS for “ << file << endl ) • Block of debugging If( BESDebug::IsSet( “nc” ) ) { list<string>::const_iterator i = _somelist.begin() ; while( ; i != _somelist.end(); i++ ) BESDEBUG( “nc”, ““ << (*i) ) }

48. BES Debugging Most BES classes derived from BESObj, and some libdap class derived from DapObj • You can dump BESObj instances and DapObj instances to ostream • virtual void dump( ostream &strm ) ; • E.g. DAS &das ; BESDEBUG( “nc”, “DAS = “ << endl << das ) • Allows for indentation as well • Register context within your module: BESDebug::Register( “nc” ) ;

49. BES Debugging - dump method example

50. Hello World • cd src/bes/hello_world • Autoreconf -fiv • ./configure –prefix=$prefix • make; make check; make install • NB: For this software, you need to do this • besctl start • bescmdln [-h localhost -p 10022] • say hello to world;