Dynamic Data Exchanges with the Java Flow Processor Presenter: Scott Bowers

1. Date: April 25, 2007 Dynamic Data Exchanges with the Java Flow ProcessorPresenter: Scott Bowers

2. History of Java Flow Processor (JFP) • Developed in 2004 for NJDEP to facilitate dynamic data publishing and dynamic data submittal services • Used at New Jersey DEP to support • Facility Registry System (FRS) submittal and query services (production), • RCRA Handler and Permitting submittal and query services (production) • Used at Mississippi DEQ to support • FRS submittal and query services (testing only) • Use at Kentucky DEP to support • RCRA Handler, Permitting, and Corrective Action (testing only) • This utility is freely available to any Exchange Network partner!

3. What is the Flow Processor? The Flow Processor: • Is a generic Java application that knows nothing about specific Data Flows. • Is aware that data is retrieved from a database and needs to be formatted as XML. • Is dynamic, allowing SQL Selects and output XML to be changed without modifying the core Java code. • Is extensible, allowing additional Java extensions to be created and assigned to data flows. • Can export or return data (submit to CDX or save to permanent storage) in a variety of formats (XML, ZIP, or TXT). • Can be invoked with or without a Node

4. What is a Flow? • The Flow Processor operates on an entity called a Flow • A Flow is identified by a unique name • The only mandatory component of a Flow is the model • The model consists of a primary SQL query, and • Zero or more detailed queries • The remaining components of a Flow are optional • Zero or more PreProcessors (Java class) • An XML Translation (XSLT) • Zero or more ExportManagers (Java classes)

5. Export Mgr Simple Generic XML Submittal to CDX XML (SQL) JFP Source XSLT XSLT Temp XML Execute SQL Get Flow Config Data Generate XML From SQL Transform Export CDX Final XML JFP Step *Execution is from Left to Right

6. <address_info> <contact_id>100</contact_id> <address>100 Maple Ave.</address> <city>Detroit</city> <state>MI</state> <zip>11111</zip> </address_info> <address_info> … </address_info> <address_info> … </address_info> <phone_info> … </phone_info> <phone_info> <contact_id>200</contact_id> <home>717-555-1211</home> <cell>717-555-2212</cell> </phone_info> <phone_info> … </phone_info> <region_info> ... </region_info> <region_info> ... </region_info> <region_info> <contact_id>300</contact_id> <region>Southwest</region> </region_info> Data Retrieval Processing: Contact Example Primary Query select contact_id “id” from Contacts Where dept=‘Sales’ Detailed Query 1 select contact_id, address, city, state, zip from Address Where contact_id=#id# Detailed Query 2 select contact_id, home, cell from Phone_Numbers Where contact_id=#id# id 100 200 300 Detailed Query 3 select contact_id,region from Sales_Regions Where contact_id=#id#

7. <get_contact_data> <address_info> <contact_id>100</contact_id> <address>100 Maple Ave.</address> <city>Detroit</city> <state>MI</state> <zip>11111</zip> </address_info> <address_info> … </address_info> <address_info> … </address_info> <SalesForceDataService version="1"> <SalesContact id="100" region="Midwest"> <ContactInformation> <Address>100 Maple Ave.</Address> <City>Detroit</City> <State>MI</State> <ZipCode>11111</ZipCode> <PhoneNumbers> <Number type="Cell">717-555-3456</Number> <Number type="Home">717-555-1278</Number> </PhoneNumbers> </ContactInformationn> </SalesContact> <SalesContact id="200" region=“Midwest"> ... </SalesContact> <SalesContact id="300" region=“Midwest"> ... </SalesContact> </SalesForceDataService> <phone_info> … </phone_info> <phone_info> <contact_id>200</contact_id> <home>717-555-1211</home> <cell>717-555-2212</cell> </phone_info> <phone_info> … </phone_info> XSLT <region_info> ... </region_info> <region_info> ... </region_info> <region_info> <contact_id>300</contact_id> <region>Southwest</region> </region_info> </get_contact_data> Transformation Step: Contact Example Temporary XML from Data Retrieval Final XML Output Post Transformation

8. Extending the Java Flow Processor • The Flow Processor can be extended by developing custom PreProcessors and ExportManagers classes. • At runtime, the Flow Processor Engine uses Java Reflection to instantiate a PreProcessor or ExportManager class. • A Java class that acts as a PreProcessor must implement the Java Interface FlowPreProcessManager. • A Java class that acts as a ExportManager must implement the Java Interface FlowExportManager. • Several Java extensions are included in the JFP: • DefaultPreProcessManager – Issues zero or more INSERT or UPDATE statements defined in the Flow Processor Settings database tables. • CDXExportManager – Submits an XML payload to a URL. • FileExportManager – Writes the resulting output to a flat file.

9. Flow Processor Complete Processing Steps

10. Deployable Components • The Java Flow Processor engine, default Pre-Processors, and Export Managers are packaged into a single Java Archive file (cgi-flow-process.jar) • It requires several third-party Open Source Java libraries, most notably: • iBATIS – Provides the data access layer from the source database and SQL mapping utilities • Apache – Several Java libraries provide support for XML parsing, XML Transformation, Connection Pooling, and Web Services (AXIS) • Log4J – Provides logging service

11. Invoking the Flow Processor • Invoked from a DOS command line or Unix shell script using the FlowProcessorCmdLine class. • Accepts just one configuration file to configure the Flow Processor • Called within existing Java applications using the FlowProcessor class directly. Two public methods are available: • performSubmit – This requires only a Flow name and returns a message of the resulting operation • performQuery – This requires a Flow name and a Java Map of potential query arguments. This returns the resulting byte stream from the JFP engine.

12. Node query() FlowProcessor performQuery() JFP Engine XML Byte Stream Dynamic Node Query Services with JFP Node Functional Specification 1.1 FlowProcessor class • <message name='Query'> • <part name='securityToken' type='xsd:string'/> • <part name='request' type='xsd:string'/> • <part name='rowId' type='xsd:integer/> • <part name='maxRows' type='xsd:integer/> • <part name='parameters' type='typens:ArrayOfstring'/> • </message> • <message name='QueryResponse'> • <part name='return' type=’xsd:string'/> • </message> public byte[] performQuery( String flow_code, int start_row, int max_rows, Map arguments)

13. Dynamic Node Query Processing Steps • Convert Node Functional Specifications 1.1 WSDL into Java code (e.g., Node11 class) • public String query(String securityToken, String request, String startRow, String maxRows, String[] parameters) • Modify Node11’s query() method to retrieve an instance of the FlowProcessor class • Either use the incoming request parameter as the Flow code or translate the incoming request into the appropriate Flow code • Convert the incoming parameters array into a Java Map • Call the FlowProcessor’s performQuery() method with correct Flow code and Map of incoming parameters • Convert the returning byte[] into a String and return this as the result of the Query method

14. Other Output Possibilities with JFP • Using Apache Formatting Object Processing (FOP), stylesheets can transform XML into PDF • SQL Reports can be built and stylesheets can transform output into HTML reports.

15. Questions?