The bottom-up approach: Challenges in the production of statistical grid data

1. The bottom-up approach:Challenges in the production of statistical grid data Rina Tammisto European Forum for Geostatistics, Workshop 1- 3 October 2008 Bled, Slovenia

2. Challenges in... • Production environment • Production methods • Quality

3. ... and relating to... • Spatiality of grids • Spatiality of statistical data • Metadata descriptions of geostatistical data • Use, instructions and distribution …from the point of view of Finland • Georeferenced data and the production of grid data in Finland

4. 5. 6. 1. 2. 3. 4.

5. 1. Challenges relating to spatiality of grid data - handling the grid net • Co-ordinate systems, projections • Standards (national/international) • National practices • Grid cell sizes • User needs • Standards • Scalability! • Practical challenge: • Enormous file sizes, difficult to handle

6. Grid net... • Essential qualifications of a grid net: • A grid net should be spatially unchanged over time. • The spatial characters should be equal to spatiality in the statistical data. • Regional classifications in a grid net: • Choice of the substantial classifications of all classifications (user needs!) . • Annual (or other) updating of regional classifications.

7. 2. Challenges relating to the spatiality of statistical data • The spatial attributes of statistical data should be equal to spatiality in the grid net. • The precision of the spatial information of statistical data. • Appropriate workflows handling the spatial information of data from preparation to storing phase. • Potential workflows in producing grid data from stored data: • From points to grids, always starting from the beginning. • Use of aggregated data • semi-finished data in the databases • ready-made grid data sets.

8. Statistical data... • Scalability • Data protection directs the choice of scale (enough data for purpose of use). • User needs vs. data protection • conflict between the user needs and data protection • scalable threshold values (?)

9. 3. Challenges relating to metadata descriptions of geostatistical data • Metadata standards • Spatial data (ISO 19115) • Statistical data - ? • Combining these two - ? • Lack of specifications for describing geostatistical information • GeoSTAT-project, subproject 3!

10. 4. Challenges relating to use, instructions and distribution • In Finland, data sets are often expensive - depends on the value of grid data for the customer. • Fewer work phases might curb production costs. • Grid data are abstract. Flexible possibilities to combine them into other regions. • USER NEEDS vary a lot. • Direct use from servers (WFS, WMS)! • Still, a GIS application is needed for use (mostly). • Easy-to-use applications!

11. 5. Georeferenced data and the production of grid data in Finland • Data from registers: • Population register, Register of buildings and dwellings • Use of geodatabase (MS SQL Server + ArcSDE) • Use of data warehouse (MS SQL Server ) • Buildings at the core • Grid data production: • Grid net updated annually • Intensive use of SQL, SAS, ESRI ArcGis

12. X,YID Geodatabase Data warehouse Selection, aggregation Deduction Grid net 250 m Database tables aggregated tables (250 m) Joining Creating a view - selection - data security Spatial database view Export SHP (personal GDB, XML file…)

13. Challenges at Statistics Finland - production • Many applications involved (SQL, SAS, AG) in grid data production • Which of the production workflows is the most qualified? • How to correctly split the workflow between the applications in order to: • avoid errors (data at this level is so precise that even the tiniest errors are visible) - quality targets! • make production as effective as possible in every way • make production as homogenous as possible (repeatable)?

14. Quantative quality elements: Completeness Logical consistency Positional accuracy Temporal accuracy Thematic accuracy - accuracy of descriptive attribute data Non-quantative quality elements: Purpose Use Lineage history of a dataset 6. Quality standard for spatial data (ISO 19113)

15. Defining the quality of statistical grid data • Quality from the point of view of users • Fitness to purposes of use • data content • spatial focusing of data • accuracy of data is sufficient for the use purposes • Quality from the point of view of producers • Quality=Data are produced “as well as possible” from the point of view of data qualifications • Quality =Data are produced strictly according to existing standards • Quality = Data are produced according to the defined target state

16. Guide,define STANDARDS LEGAL REGULATIONS AND RESTRICTIONS TARGET STATE adjust The definition and description of the target state USER REQUIREMENTS guide, describe Several parallel requirements: -> intended use of data QUALITY instruct,describe USE PRODUCTION Descriptions of data Instructions for use Description of the method

17. Finally.. Our forthcoming project should give us a STRUCTURE for handling this never completed process.

18. How to respond to the challenges? • User needs define quality. • User needs are in focus when setting targets. • Standardized production • Predefined minimum standards for quality(beyond the changes in production environment or methods). • Instructions for use and descriptions of the purpose of use. • An effective data delivery and use possibilities (Google Map, Google Earth,…).

19. Thank You for Your attention!