1 / 14

Node Lessons Learned

Node Lessons Learned. James Hudson Wisconsin Department of Natural Resources. Nodes Evolve. Phase 1: Getting the Node Working Done Phase 2: Making Data Flow Done Phase 3: Current Reproducible Implementation of Multiple Flows Incoming Data Flows Node Management and Administration

thuong
Download Presentation

Node Lessons Learned

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Node Lessons Learned James Hudson Wisconsin Department of Natural Resources

  2. Nodes Evolve • Phase 1: Getting the Node Working Done • Phase 2: Making Data Flow Done • Phase 3: Current • Reproducible Implementation of Multiple Flows • Incoming Data Flows • Node Management and Administration • Phase 4: Future • New Standards (attachments, authentication, WSDL changes, BPEL and orchestration) • New Technology (Enterprise Service Bus, etc.)

  3. Wisconsin’s Perspective • We were Not in the Beta or in Node 1.0 • “Early Adopter,” right after those states • Built our own node (there were no DNCs) • Built our own FRS flow (no FCDs) • Continue to enhance our processing • The node is key technology for our agency and state

  4. Challenges • Node Management • Environment Changes • Designing for a Reproducible Process • Being able to implement necessary changes • Who has the correct data?

  5. The Node Management Challenge • The Node Specification describes the Web Services • The Flow Configurations and schemas describe the Data and Requests • But there is no standard for what information to store about your node • Or how to manage that node • Solution so far • We built our own tools, as have others • Can this be Standardized?

  6. The Environment Challenge • So far, Wisconsin has been through 2 version changes in the Oracle Application Server • We’re moving to Websphere since that seems to be the enterprise choice for the state, and will give us a better service level agreement • Any change in environment takes time and effort and can get in the way of other work • Solution: invest the time

  7. The New Flows Challenge • We have a stable node • We want to implement at least 8 flows in the next 2 years. And possibly up to 25. • How do we make those flows easy to build, easy to change, and high performance? • Our strength is the database • Data is there already • So are our skills • And our processing horsepower

  8. Wisconsin’s Version 1: Java • Configured Node using XML files • Logged to files on the Application Server • HTTP and SSL on the App Server • Used a utility in Java to query Oracle and extract the XML • Converted relational to XML in Java • XSLT in Java • Schema Validation in Java • ZIP in Java • DIME in Java

  9. What We Learned from Version 1 • Application Server was hard for our staff to administer • Java performance was slow and required too much memory for large operations (XSLT on 100MB of XML, for example) • Needed Expert Java Developers • New flows, especially incoming flows, would require writing too much code

  10. Wisconsin’s Version 3 • Node functions still in Java • Configuration and logging stored in Oracle • SOAP, ZIP, DIME all in Java • May move most validation to CDX’s web services, based on the presentation yesterday • Flows nearly all in Oracle • Java program stores the Request and calls a database package • Results come back to the Java program for packaging and shipping (header still built in Java, for now)

  11. How Does the Flow Work? • For each new flow: • Convert Schema to Oracle Types • Map to existing database tables by Object-Relational Views (the hard part) • We use database built-ins to • run query and convert to an XML document • parse using Document Object Model • XSLT • store the results

  12. What Does This Gain Us? • Performance: FRS Solicit, consolidated schema, 40,000 facilities: • 3.5 hours on production server with version 1 • 25 minutes on development server, version 2 • Standard Steps for building a new flow • Less need to Redeploy the Java app • Better skills match • Better able to handle two-way flows • Less dependent on the application servers

  13. “What Data is Best?” Challenge • This has been an issue since states started reporting to EPA • Can users trust the national data? • Can they trust the state date? • How can they tell why the data are different? • With the Exchange Network, we may be able to finally deal with this problem

  14. Final Comments • The technology is getting much more stable • We’re dealing with data issues, like multi-agency consistency, that we could never address before • It won’t be cheap or routine for a while, and you’ll need programmers/contractors • But we’re making a difference

More Related