1 / 27

NeSS Data Exchange 2.0

NeSS Data Exchange 2.0. Timeline for NeSS Data Exchange. Started as a pilot called Cafeteria in 2006 Rejuvenated by joint initiative with CLG 2007 Beta April 2008 Release 1.0 December 2008 Release 1.1 April 2009 Release 1.2 November 2009 Release 2.0 February 2010

bella
Download Presentation

NeSS Data Exchange 2.0

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NeSS Data Exchange 2.0

  2. Timeline for NeSS Data Exchange • Started as a pilot called Cafeteria in 2006 • Rejuvenated by joint initiative with CLG 2007 • Beta April 2008 • Release 1.0 December 2008 • Release 1.1 April 2009 • Release 1.2 November 2009 • Release 2.0 February 2010 • Eventually to be replaced by WDP API

  3. Cafeteria • Pilot system built in 2006 • Web service layer over NeSS database • Generic NeSS-ML output XML • No “discovery” • Used by iGather (Scisys) • Left on hold until decision to commence NeSS Interoperability project which was part-funded by CLG

  4. CLG’s Future Vision: The “Data Web” Inspectorates OGDs DCLG DCSF ONS Audit Commission Places dB CentralServices Reg Obs GSi Internet Methods System to System LA LA LA LA LSP LSP LSP LSP Hubs 4 Author : M Stone v 1.6 30/01/08

  5. NDE’s Role • To supply CLG’s Data Interchange Hub • To support other Local Information Systems • National usage secondary consideration • Web 2.0 apps also secondary consideration • Service can be used live - “hunter” • Or to accumulate data - “gatherer”

  6. Hunter Gatherer

  7. DB Browser Web Site Data Access Component Fig. 1 Web Site Browser AJAX NDE Web Service Data Access Component DB Spread Sheet Excel Macro Other Java .Net etc. Fig. 2 Web Service Web Site vs. WebService

  8. XML Formats • Looked at SDMX – too complex, problems with geography, output hard to consume • Also considered keeping NeSS-ML but it was considered too abstract and generic to be easily consumed • Went with LGDX (was PIE) to be consistent with CLG hub – also simple and rigid so easy to consume. Only downer is it cannot support cross-tabs. • Retained NeSS-ML for internal use and potential transform to other formats in future.

  9. Discovery • Half of NDE is the Discovery Service • This allows you to ask questions of the NeSS database. Often this will be a string of queries…. • Get list of subjects • User selects “Health” • Get datasets for subject “Health” • User selects “General Health UV20” • Get variables for dataset UV20 • User clicks i button next to “Not Good Health” • Get metadata for this variable

  10. Delivery • The other half is the Delivery Service • This delivers the data payload as a self-documenting chunk of XML. • Initially this required a complex request to be built up from discovery calls. • Simplified in Release 1.1 • Initially only LDGX format • SDMX and RDF added in Release 1.2

  11. Clients • ONS supplied sample clients to show how the service can be used with different languages • Some interest from commercial companies to supply “middleware” • Access initially via SOAP only (REST-like version to follow) • Authenticated access via manual registration only. Valuable at first but not seen as so important now.

  12. Sample Client (Excel and SVG)

  13. Pros and Cons of NDE • POWERFUL – everything the NeSS website can do you can do with NDE, including useful geographical functions • COMPLEX – big manual, many operations, understanding of NeSS database required • RELIABLE – 99.6% uptime • INCOMPATIBLE – with some newer systems, notably Adobe Flex and Ruby

  14. Where are We Now? • NDE has had generated only moderate interest so far, and there are currently no live applications using it as far as we know (though SASPAC beta looks promising). • Versions 1.2 and 2.0 look to redress some of the “Cons” and provide alternative output formats (corporate requirement, not from NeSS) • Small budget for improvements but committed to continued support for foreseeable future. • NDE is proving to be a useful toe in the water for the WDP API that will eventually replace it.

  15. What’s New? • VERSION 1.2 • Adobe Flex support (+demo) • SDMX output • RDF output (experimental) • VERSION 2.0 • Lite Service

  16. Adobe Flex • Premier Data Visualisation Tool • NDE 1.1 does not have the correct configuration to allow Flex applications to call it. • Changes made to 1.2 to make it work. • Demonstration “dashboard” application supplied. • Uses SOAP service, will supply a REST-like version with 2.0

  17. Client Screenshot

  18. SDMX • International Data Interchange Format • Being adopted by ONS, NOMIS and many overseas organisations such as Eurostat • Can comprehensively model all data • Flexible and Complex • Generated on-the-fly by NDE • Message contains definitions and data • Harder to consume than LGDX Sample Output sdmx_sample2.xml

  19. RDF • Building block for the Semantic Web • Triples consisting of subject, predicate and object • Format favoured for Tim Berners-Lee’s UK Public Data Project • Generated on-the-fly by NDE • Experimental format based on SCOVO • Could supply complete geographic hierarchies like OS Parishes

  20. RDF Example • Alan Pardew (subject)is manager of (predicate)Southampton FC (object). • Markus Liebherr (subject)is chairman of (predicate)Southampton FC (object). • Using this method everything on the www is interlinked • Good for metadata • Possibly too verbose for large datasets – NeSS has about 10,000,000,000 triples. Sample file ness_transformed_to_rdf.xml

  21. Lite Service • Simplified, fewer operations • REST or SOAP • Low security • Optional registration • Aimed at web 2.0 and other lightweight users • Possibility for reduced functionality

  22. Simplified Operations • Hide some of the complexities of the NeSS database • Cut down the perceived number of operations by adding a few more! • Split into “Core” and “Advanced” operations • Rationalise the area search into a single operation • Make the user documentation less daunting

  23. SOAP Web Service • The Simple Object Access Protocol • Standard for web services for many years • Usually operates via an HTTP POST • Operations defined via a WSDL document • Many languages such as Java and VB can use the WSDL to create objects • SOAP can also be used for AJAX applications via XMLHTTP with raw XML but it is clunky

  24. REST Web Service (HTTP GET) • Not RESTful by the textbook definition, but calls are made via a simple URL, e.g. • http://www.neighbourhood.statistics.gov.uk/interop/NeSSLiteDiscoBindingPort/GetAreaChildren?Areaid=565457 • Can only accept simple parameters (strings and numbers) • No auto-generation of objects from WSDL • Preferred to SOAP for many modern web platforms

  25. Discovery Endpoint Request Basic Authentication HTTPS SOAP Header WS-Security UserID Password OPERATION SOAP Body BOM NeSS Login Module XRender Request SOAP Header WS-Security UserID Password HTTPS Delivery Endpoint Basic Authentication SOAP Body OPERATION Security: Full Service

  26. Request Lite Discovery Endpoint HTTPS OPERATION URI BOM XRender Request HTTPS Lite Delivery Endpoint URI OPERATION Security: Lite Service

  27. NDE Lite - Other • Despite no need to register, users can still sign up for notifications and tech support • There will be no loss of functionality except for the complex getDataCube function • Current plan is to run NDE Lite as a separate application in parallel with the current service. • Turn off old service once users have had plenty of time to migrate

More Related