Raptor Technical Details

1. Raptor Technical Details

2. Outline • Workshop structured by Raptor workflow • Raptor Event model. • ICA log file parsing • ICA/MUA event storage • ICA event release • MUA batch event persistence • MUA statistical processors • WEB overview • WEB and MUA hookup and operation • Live walkthrough of creating a new event time

3. Raptor Event Model • Raptor uses a concrete Java (POJO) event model e.g. each event is a ‘real’ class (with appropriate hierarchy).

4. Raptor Event Model

5. Raptor Event Model • To create a new event class, you *must* currently do the following. • Check-out the current raptor-information model from svn. • Create a new class in the event package • Add necessary fields • Create a toString method (for debugging) • Create a hash and equals method (so duplicates can be understood – business key). • Configure hibernate xml file for new type. • Use maven to recompile the library, rename to same as existing raptor-information-model and replace in lib dir of ICA, MUA and WEB.

6. Log File Parsing • Event-parse specifies a number of BaseEventParsers. Each currently a log file parser. • Each parser must determine from line filters which lines are of interest. • Each parser has associated format, which gives mappings from values in the log file to the internal model field name.

7. Log File Parsing Flow

8. Event Storage • In memory on ICA and DB backed in MUA. Both share similar flow…but. • ICA uses a slightly different storage semantic. Incremental checks for efficiency, whereas MUA does not make assumptions about temporal ordering, just checks for existence and if not found stores.

9. Incremental Event Storage

10. Persistent Event Storage

11. ICA Event Release • Releasing events to configured MUAs (more than one, although small technical bug that needs fixing) • Each MUA is an endpoint and accepts a particular type of event(s). • Different release policies to different MUAs • Number of events • Elapsed time • Or combination

12. ICA Event Release • Events release is called every 30 seconds. • High level locks exist to stop release from running if capture (or any other operation) is running. • Applicable events are sent to each endpoint individually. • Events filtered by time • Events filtered by type • Events filtered by resource category (internal/external) • Events sent over as SOAP messages over HTTPS using apache CXF • Events class marshaled in and out of XML using Aegis binding.

13. ICA Event Release

14. MUA Batch Event Parsing • ICA uses an in-memory Set to hold events. If ICA dies, so do the events it holds. But restarting the ICA reparse and re-send all events. MUA will then discard on the hash id of the event. • MUA uses a persistence layer to map Events to tables in the db. • Each event requires a Hibernate ORM configured. • Hibernate then maps from the object through to the correct SQL dialect on the configured database.

15. MUA Batch Event Parsing • Events are received from the ICA through its exposed SOAP endpoint. • Events are stored asynchronously via a background worker thread. • Each event is passed through the attribute expansion engine before storing. • Events are stored through a similar event handler as the ICA – only configured to talk to the DAO layer.

16. MUA Batch Event Parsing

17. MUA Statistical Processors • MUA stores all the statistical units and processors necessary for computing stats. This does not reside within the Web component. They are like templates. • Statistical-units-custom stores user configurable (through the web UI) stats. • Statistical-unit-system stores the stats the web interface calls upon to do the dashboard statistics.

18. MUA Statistical Processors • New stats can be configured in the XML, and the MUA needs to be restarted for them to take affect. • MUA compiles a capabilities object with those statistics loaded and passes this to the Web component so it knows what is available. • Each statistic has a number of properties for series definition, display and post processors.

19. MUA Statistical Processors • Post processors can be attached to a statistic to perform certain functions e.g. add SAML org name, or remove rows from result. • Each processor is defined in the statistical-processors.xml file. • Processors are instantiated and added to a statistic whenever the stat is created or updated.

21. MUA Statistical Processors • Most heavy lifting on the database layer e.g. groupby and temporal queries • However, results are loaded into the MUAs memory to: • Allow post processors to act on them • Construct a graph model so that the web interface can construct a graph image from them. • This is one of the primary reasons the MUA eats memory (not really on the event storage lifecycle). • The Web interface also takes some time and memory to construct a graph image – using JFreeChart.

22. Web Overview - Technologies • The web interface is a Java servlet, running spring web flow, Java Server Pages, and Java Server Faces for UI components. • Also used Richfaces components • Follows Model (Java Pojo), View (JSP/JSF), Controller (spring webflow) paradigm. • In reality, view components get decoded to HTML and Javacript (lots of) which makes calls and (Ajax or full page refresh) to the Java Servlet (which runs on Jetty).

23. Web Overview - Functions • Can attach to any MUA (providing keys are in place) • Runs set of dashboard statistics • Can perform batch uploads to the MUA (limited file size due to non-streaming transport). • Can invoke any of the retrived MUA statistics • Converts the graph model sent back by the MUA into a graph and table • Can convert the table and or graph into a PDF, CSV or Excel file format.

24. Web Overview – Interaction with MUA • When attaching to an MUA (when it starts, or the user requests it), Web asks the MUA for its capabilities. These are: • Statistical units • Statistical processors • Event information e.g. no. of events per type. • Suggestion values • Resource Information.

25. Web Overview – Interaction with MUA • Capabilities information is used throughout the UI e.g. • In the setup page to show resource classifications and event information • In the graphs page to list possible statistics and their values • Suggestion values are used in the drop downs for filters • The dashboard stats creations (explained next)

26. Web Overview – Interaction with MUA • The definition of what is run by the dashboard page is stored in the web configuration (dashboard-statistics.xml). • This stores a list of what type of statistics to run, over what time period and for what event type. • The actually statistic it calls in defined on the MUA in the statistical-units-custom.xml file. • So its almost a meta-template for a statistic. • These can be amended or turned on and off there. • Some stats can take a long time to run e.g. unique users.

27. Web Overview – Interaction with MUA • Graphs page shows those statistical units retrieved from the attached MUA. • Clicking on the statistic sends an invocation request to the MUA using the statistic name as the stat to invoke. • Expanding the statistical parameters allows you to alter the statistic. By clicking on update it sends the entire set of statistical parameters back to the MUA so the MUA can update it. It then also sends an invoke request to perform the statistic.

28. Web Overview – Batch Upload • Web loads each specified file into an in-memory object. • This object is then send (via CXF, SOAP) to the MUA. • The file can not be streamed due to the SOAP binding raptor uses. • Hence a limit on the number of files.