Monitoring the acquisition process by web widgets Leonardo Tininini and Antonino Virgillito ISTAT

1. Meeting on the Management of Statistical Information Systems(MSIS 2011) Monitoring the acquisition processby web widgets Leonardo Tininini and Antonino Virgillito ISTAT Luxembourg, 23-25 May 2011

2. Outline • Motivations • The web widgets • The "General picture" widget • The "Regional thematic map" widget • The "Latest news histogram" widget • Enhancing performances by materializing data • The widgets' software architecture Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

3. Motivations • The web is increasingly becoming a fundamental means for supporting several phases of the statistical data life cycle, particularly data acquisition and dissemination, however... • little interest has been devoted so far to use the web for publicly monitoring the acquisitionprocess • Objectives: • providing, as far as possible, fresh data • do not disturb... (the acquisition process itself) • make the components “embeddable” in any web page Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

4. Web widgets • A widget is a small web component that can be freely “embedded” into any web page (e.g. a blog), without requiring specific technical knowledge • Widgets can be composed and organized in personalized dashboards according to the user's specific needs and preferences • The widgets described here display some key data about the 2010 Italian Agriculture Census acquisition process Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

5. The “General picture” widget • Shows the numeric values of 5 main aggregates: • the number of operators • the number of interviews completed by operators and recorded in the Census Management System • the number of questionnaires (fully) compiled by enterprises by using the web • the number of questionnaires returned (compiled) by operators by the web • the number of questionnaires under compilation (i.e. only partially compiled) on the web by enterprises. • Accepts an additional parameter to select the territory of interest (Italy / a specific region) • Refreshed every minute Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

6. The “General picture” widget (snapshot) Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

7. The “Regional thematic map” widget • Same 5 aggregates as those in the “General picture” widget, each selectable by its own tab • Aggregates are represented on the map of Italy using classes of values • Exact aggregate values, along with names, can be obtained by rolling over the region on the map • Each region's color is faded in proportion to the value of the specific aggregate for that region • The maps are built using the freely-available Google Interactive Chart Tools API (specifically, the Geo Map component) Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

8. The “Regional thematic map” widget (snapshot) Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

9. The "Latest news histogram" widget • Shows the time distribution of the questionnaires compiled over the last 24 hours • Both enterprises and operators • the fraction corresponding to enterprises is depicted in darker green • the one corresponding to operators in lighter green • Details can be obtained by rolling over some sections of the widget • the roll-over on each bar displays the details of the two component values (enterprises vs. operators) • the roll-over on the two legend items displays two different histograms, corresponding to questionnaires compiled only by enterprises or operators, respectively Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

10. The "Latest news histogram" widget (snapshot) Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

11. How to show the enterprise-only histogram Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

12. Enhancing performances by materializing data • Data displayed by widgets are expensive to be computed (counts on several hundreds of thousands of records) • Unpredictable number of users (due to free embedding) • Potential performance degradation for both: • the operators and enterprises working on the Census Management and Acquisition System and • the widget users • This potential degradation has been avoided by adapting two techniques, commonly known as view materialization and query rewriting, widely used in data warehousing environments • Displayed data are obtained by using (and possibly further processing) a few pre-computed data, instead of aggregating data from scratch Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

13. Materialized views and query rewriting Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

14. The widgets' software architecture • Single Java web application, directly accessing the Census database • High scalability: minimum (and bounded) overhead on the database against unpredictable request traffic • Decoupling the access to the widgets made by the user from the access to the database (with aggregation) made by the widgets • One component (Update Job) computes the aggregates (storing them into the application memory) and is scheduled to automatically run every minute • The front-end component (widget in stricter terms) displays the aggregates stored in the application memory Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

15. High-level architecture of the web application Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

16. The widget embedding process • Each widget is implemented as a separate JSP page • The embedding starts form a single row of predefined HTML code that has to be included in the page (Embedding code) • the code produces a call to a JavaScript fragment (Embedding script) that is stored on the server and is downloaded and executed on the client browser • The execution of the Embedding script creates aframe within which the JSP page rendering the widget is actually loaded • Through this technique, the widgets can be incorporated into virtually any web site or blogregardless of the underlying technology and without any programming knowledge <script src="https://censimentoagricoltura.istat.it/CensWidget/caTable.js" type="text/javascript"> </script> Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

17. Conclusion and future works • The proposed design can be considered as a general pattern for the acquisition process web monitoring: • easily embeddable “lightweight” components • high level of scalability, by... • decoupling the acquisition data from the monitoring data provided to users • price to pay: “freshness” of the information provided • Next (more severe) testbed: monitoring the acquisition process of the Population Census • one/two order(s) of magnitude larger • the trade-off between freshness and overload will be much harder to achieve Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011

18. Questions? Tininini and Virgillito - Monitoring the acquisition process by web widgets - MSIS 2011