1 / 58

Server-Side Functions

Server-Side Functions. Contents. Background on DAP2 and Constraints Writing Server Functions: short version Programming with libdap A Real Server Function Write HelloWorld() Advanced Topics. Background on DAP2 and Constraint Expressions. DAP2 interface is via the URL

agable
Download Presentation

Server-Side Functions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Server-Side Functions

  2. Contents • Background on DAP2 and Constraints • Writing Server Functions: short version • Programming with libdap • A Real Server Function • Write HelloWorld() • Advanced Topics

  3. Background on DAP2 and Constraint Expressions • DAP2 interface is via the URL • The URL includes the • Name of the data set (/data/nc/fnoc1.nc) • Name of the desired response (DAS, …) • Optional Constraint Expression • Constraint expression chooses which parts of the dataset (granule) are returned • If the dataset has several variables, chooses among those • If the variables are arrays, subsets those • If the are Structures, chooses the fields • If there are Sequences, selects values • Also runs functions if they are available

  4. Constraint Expression (CE) Syntax • URL: http://host/dataset.<ext>?<ce> • If <ext> is ‘dds’ or ‘dods’ the <ce> is used to contrain the response • <ce> is a <projection>[&<selection>]* • <projection> • is a comma separated list of zero or more: • Variables to be returned. • Array hyperslabs • Structure fields • Function calls

  5. CE Projection Examples • Using the http://localhost:8080/data/nc/fnoc1.nc dataset: • ?u,v Chooses the variables u and v • ?u[0][0:5][0:5] Chooses u and subsamples the array returning a 6 by 6 hyperslab • Try these in a browser - you do not need to use the virtual machine’s server; you can substitute test.opendap.org for localhost • e.g.,http://test.opendap.org/opendap/data/nc/fnoc1.nc.ascii?u[0][0:5][0:5] • I used ‘.ascii’ instead of ‘.dods’ so the result is easy to see in a browser

  6. More CEs • http://test.opendap.org/opendap • /data/nc/coads_climatology.nc.dds?SST

  7. Choosing Fields • http://test.opendap.org/opendap • /data/nc/coads_climatology.nc.dds?SST.TIME

  8. CEs for data • Replace the ‘.dds’ in the previous URLs with .ascii or .dods (use ascii with a web browser or dods with an application that understands binary data) • Look at the results • Try various combinations

  9. What About Selections? • Selections are used to choose values from relational types • Try it: • http://localhost:8080/opendap/data/ff/avhrr.dat.asc?day • http://localhost:8080/opendap/data/ff/avhrr.dat .asc?day&day=19 • http://localhost:8080/opendap/data/ff/avhrr.dat .asc?&day=19 • The first constraint is a projection that chooses just the variable ‘day’ while the second constraint includes a selection clause which chooses only those entries where the value of day is 19. The third constraint returns all the fields but only those rows where day is 19

  10. But there’s more to Constraint Expressions • They can invoke functions • The functions can compute and return values • They can also return a Boolean value for use in the selection part of the constraint expression

  11. Server Functions • Functions are loaded into the Constraint Evaluator • Each handler can load its own set of functions • Functions can also be loaded by ‘function modules’

  12. Function syntax • <ce> can be one or more function calls • <function> “(“ <zero or more args> “)” • <function call>,<function call> • Functions can be composed. • Example of a simple function call: • http://test.opendap.org/opendap/data/nc/fnoc1.nc.ascii?version() • Run this example in a browser or command shell using getdap • This is a very simple function and it takes no arguments • It’s intent is both as an example and to provide information about the functions leaded into the server. jimg : ~/src/hyrax_1.8_release $ getdap "http://test.opendap.org/opendap/data/nc/fnoc1.nc.ascii?version()" Dataset: function_result_fnoc1.nc version, "<?xml version=\"1.0\" encoding=\"UTF-8\"?> …

  13. More Useful Functions • grid(): sample a Grid variable based on the values of its map vectors • geogrid(): just like grid(), but assume that the Grid holds geospatial data • linear_scale(): Scale data using y=mx+b • These are defined in the ‘functions’ module (see $HOME/src/hyrax-1.9/src/bes/functions)

  14. Function Documentation • Use ?version() to get a listing of installed functions. • Use ?<function>() to get help information specific to a particular function • Go to docs.opendap.org*

  15. Example • Use the geogrid() function to select values from the COADS Climatology data set: • ?geogrid(SST,-5,-80,-40,-50)

  16. Observations • Note that the lat/lon values from the call match those of the response • The arguments are a little odd but help is available by calling the function with no arguments • This function’s logic is pretty crude, but the same interface can be used in front of more sophisticated processing (i.e., GIS) • Calling the function with no arguments returns version information and the URL of the function’s documentation. • Note also that this made no choice about the time information in the SST variable…

  17. Selecting TIME • ?geogrid(SST,40,-80,5,-50,\”2000<TIME\”) • Try it - be careful with the quotes • The general rule for geogrid() is that the first five arguments are required (they specify the Grid variable and the top, left, bottom, right (TLBR) lat/lon box) and then subsequent arguments are relational expressions which include constants and variables which are the names of the Grid’s map vectors. • The result is the intersection of the zero or more relational expressions

  18. Writing Server Functions: short version • Write a C++ function which uses one of the three server function type signatures • Make an instance of the ServerFunction class class • Compile/link the new code into a handler • An annotated example follows…

  19. A Simple Example • Look at the libdap source file VersionFunction.cc and scan the function “function_version”

  20. Arguments. Conceptually similar to the arguments passed to main() when writing a C program Name Create a variable for the return value Set the value value Return the variable

  21. Adding the Function into the Constraint Evaluator • Constraint Expressions are evaluated using the ConstraintEvaluator C++ class • The ConstraintEvaluator::add_function method is used to add a function and bind that function to a symbolic name used in the constraint expression • From the ce_functions.cc source file, find the function register_functions(ConstraintEvaluator &) • To add a function and bind it to a name use add_function() • ce.add_function("version”,function_version);

  22. Functions are added to the ConstraintEvaluator • Look in the ConstraintEvaluator.cc and ce_functions.cc source files to see how the base set of functions are added to the default constraint evaluator • Note that ‘register_functions’ is a function defined in libdap’s ce_functions.cc file

  23. Functions can be added to handlers, too • In the handler’s specialization of the RequestHandler class • This class contains the methods that build the response objects • Add an explicit call to add_function to the method that returns data: • Bool NCRequestHandler::nc_build_data(BESDataHandlerInterface & dhi)

  24. Programming with libdap • The library has classes for the different datatypes (Byte, Int16, …, Structure, Grid) • Each of those are children of BaseType • The common OO pattern of using a generic parent class to pass more specific instances is used often by libdap • The DDS object in libdap is used to make all of the DAP responses (although there is a ‘DAS’ object in the library for ‘historical reasons’). • The DDS holds all of the variables of the data set • The variables in the DDS contain the attribute information but hold no data until either data values are read or until just before the server runs the send_data() method.

  25. Reading and Sending Data • Three pieces are involved: The handler; the BES and libdap • First the handler builds an instance of DDS that describes the entire dataset • Then the BES calls uses a ‘Transmitter’ to run methods in libdap • Finally, libdap parses the constraint, evaluates the constraint and serializes the result

  26. What the handler does • Initializes some data objects that are specific to the handler • Sets the ConstraintEvaluator instance • Reads flags (aka ‘keys’) for the handler from the bes.conf file • Uses the BESDataHandlerInterface to • Get a pointer to the DDS object • Populates that DDS with variables • Adds attribute information How the handler populates the DDS is dependent on the underlying data store. A handler for HDF4 uses HDF4 API calls while a handler for SQL databases uses a very different kind of interface See bool HDF4RequestHandler::hdf4_build_data(BESDataHandlerInterface & dhi) in HDF4RequestHandler.cc

  27. What the BES does • The BES uses a ‘Transmitter’ to: • Pass the DDS object built by the handler to libdap • Pass the ConstraintEvaluator instantiated by the BES to libdap • Pass the Constraint Expression from the URL to libdap See class SendDataDDS in BESDapTransmitter.cc

  28. What libdap does • Parse the constraint • If there are function calls in the constraint, then evaluate them and store the results in the DDS • Serialize the DDS – this is the step where the response is actually sent back to the client See void ResponseBuilder::send_data in libdap:ResponseBuilder.cc

  29. More about libdap’s role • The variables held by a DDS may contain data (but they do not have to) • Data values are read into the variables in a DDS using read() methods which are specialized for each handler. • For example, the NCArray::read() method for the netCDF handler is different than the HDF4Array::read() method in the HDF4 handler • By convention, the read() method tests a ‘property’ called ‘read_p’ to see if the read() method has already been run. This keeps software from reading data for a variable more than once.

  30. The read() method: An Example • Implementation of HDFArray::read() • Note that the ‘helper’ function sets the ‘read_p’ property when data are read correctly. …

  31. How Constraints are Evaluated • When libdap makes the data response, it first parses the constraint expression (see parse_constraint() in the class ConstraintEvaluator). • This parses the projection information and marks the variables in the DDS which are part of the projection • The ‘send_p’ flag is used for this purpose • Each variable has a send_p flag and the send_p() accessor and set_send_p() mutator can be used to access and alter its value. • It binds any hyperslab operators to the Array variables • And it parses any functions, recording them for later evaluation • While the evaluator sets the send_p flag based on the parse of the projection information, other code which has access to the variables can also work with this flag (server functions are one example).

  32. Aside: What happens when there are no server functions? • This is actually the typical case… • The serialization code iterates over all of the top-level variables and • For each variable where send_p is true, • Reads the data values using read() if read_p is not true • Uses XDR to encode the data and • Writes that encoding to the output stream

  33. Server Functions can Expect • The DDS will: • Contain all of the variables in the dataset • Each variable will have its attributes • There will be no data in the variables • Variables in the project will have the ‘send_p’ property set – but that does not extend to variables passed to the function as arguments

  34. Other steps you’ll need to take • Once you’ve written a CE function… • Compile it • Write the software to add it to an instance of ConstraintEvaluator • Recompile/link the relevant code • Restart the server

  35. Where should I put my functions? • You can bundle them with libdap - as the base set of functions are • You can add them into a handler • The FreeForm handler includes a fairly large set of functions - open that software and look for the mechanism used to load them

  36. Different ways to load CE Functions • They can be added by the ConstraintEvaluator constructor in libdap or by a handler as is the case with FreeForm

  37. Overwriting Functions • The ConstraintEvaluator::add_function method maintains the function name/code binding so that newer entries with the same name take precedence. • Thus a specific handler can override the definition of one of the base-set functions

  38. A ‘Real’ Server Function • Look at function_linear_scale in ce_functions.cc • Synopsis: • The function does some fancy processing with arguments - if arguments are passed it will use those values, if not it will look at the attributes for the scale factor (m) and add offset (b) • The scaling operation is slightly different for Grids and Arrays • Because the variable name was passed in via a function call list, the expression parser did not set the send_p flag - the function sets it • Then the function reads the data values from the data set into the variable

  39. Argv[0] is first argument to function and is the variable to scale. Here we use the BaseType method Is_vector_type() to determine that argv[0] is an Array and then use a C++ trick to cast that BaseType pointer down the class hierarchy to an Array. This makes it easier to call methods specific to the Array class As the comment says, if the Array is really a Grid map variable, be sure to use Grid::read() to load data values into variable. If not, just call Array::read()

  40. Extract_double_array() is a helper method that reads numerical values from a DAP variable and loads them into a dynamically allocated double array. It’s defined in the ce_functions.cc source file The Array::length() methods returns the number of elements in an array Y=mx+b

  41. More… • Then the values are copied from the variable in the DDS to an Array of doubles • This array will hold the result of the scaling operation • The values are scaled • A new DAP array of Float64 is created and the scaled values are stored there • The newly created DAP Float64 array is returned

  42. Create a new Float64 (the DAP type for 8-byte real values) which serves as a template for the type of the array’s values. Use the old variable’s name. Insert the new template type into the Array object Load the data values into the Array - this is the old-style value loading method. It’s better to use set_value()

  43. Write a HelloWorld Function • Write the function in the FreeForm CE functions file ff_ce_functions.cc • Recompile and relink the FreeForm handler • Install the modified handler • Test

  44. Open ff_ce_functions.cc

  45. Add the function • At this step it’s just a stub • The code should still compile - check it by typing ‘make’

  46. Now add the function’s guts • Use C++’s string class to hold the value • Create a new DAP ‘Str’ (String) variable • Load the value • Return the Str instance - Str is a child of BaseType

  47. Compile to catch any errors…

  48. Now arrange to register the functions • Use ConstraintEvaluator::add_function() • If you add this call to ff_register_functions() then you’re all set • How else could you add the function?

  49. Compile, Install and … • Compile: ‘make’ • Install: ‘make install’ • This will copy the freeform handler shared object library (aka ‘module’) to the directory where the BES expects to find it.

More Related