Presenting Statistical Data Using XML

1. Presenting Statistical Data Using XML Office for National Statistics, United Kingdom Rob Hawkins, Application Development

2. Presenting Statistical Data Using XML Supporting paper for MSIS 2005 Reference: CES/AC.71/2005/23 • Mission • Requirements and Examples • Component Design • The Future • Questions at the end please !

3. Mission Develop a Java component, which • queries a large relational database (300 million statistical values and supporting metadata) • produces a variety of statistical outputs • enables a range of user interactions with the outputs • can be integrated into any Java-based web application • performs and scales well on busy public-facing web-sites • is flexible and extensible

4. Mission Key customer: UK Neighbourhood Statistics Programme (NeSS) • Web-site to provide better information for neighbourhood renewal Also, build the component so that it can be re-used for other web projects

5. Requirements and Examples HTML tables

6. Requirements and Examples Charts

7. Requirements and Examples Thematic maps

8. Requirements and Examples Print-friendly examples

9. Requirements and Examples Metadata Linking • Requirement to access contextual metadata from items in tables, charts, and maps

10. Requirements and Examples Neighbourhood Profiles (a report per area, with multiple outputs on same page)

11. Component Design • Simple API hides internal complexity • Internal re-design can be carried out without affecting the client-code • Written as a “framework” for re-use • Stateless and not dependent on web-site that uses it • Design is suited to being encapsulated by a Web Service

12. Use of XML and XSLT • Instead of building the output directly from relational database queries, we first create the results as XML • We then use XSLT transformations to convert the XML into the required form (HTML, CSV, SVG, etc) • Re-usability of the solution is improved by use of a common, multi-dimensional XML Schema • As long as XML conforming to the schema can be produced, any data source can be used, e.g. legacy sources • The entire process is dynamic, carried out when the web-site user clicks “go” – querying the database, building the XML, running the XSLT transformation • This is all run on the server – there is no transformation taking place on the client

13. Simplified design diagram Client code Data Request (What I want) Common XML Schema Data and Metadata XRender Service Layer Process Layer Presentation Request (How I want it) XML Data and XML Cache XSLT Presentation Rules (XSLT) HTML table HTML,SVG, CSV XML & XSLT HTML, XML Chart (URL) XML Transformer XML ArcXML HTML SVG Map GIS Service PopChart CSV, etc.

14. Re-usable framework “Plug-and-Play” Design “Business” = Business-domain specific code and configuration – e.g. for NeSS requirements Business Data Access Objects turn database queries into XML Generic Objects Model and Build the Common XML Schema Generic Utility Objects e.g. for Caching, automating the transformations, building charts Generic XSLT chains Business Configuration XML used to tailor the output Business Data Sources & Stored Procedures Business XSLT chains Generic Process objects Coordinate the Transformations Environmental configuration XML Business Process Objects build the client response Business API layer Business Cascading Style Sheets Business JavaScript

15. Performance and Scalability • High-volume public web-site – peaks at hundreds of concurrent user requests • Requirement to render most output within a couple of seconds as a worst-case average • Techniques used: • Oracle index tuning and query hints • Memory caches to minimise database access. • XSLTC (Pre-compiles XSLT scripts) • Balance and optimise the work done by Java and by XSLT

16. The future • Further ongoing enhancements for NeSS • Re-use component on UK “National Statistics Online” web-sites • Share ideas (and code?) with other National Statistics Institutes ?

17. End of presentation Any questions ?