Using FME for Topographical Data Generalization at Natural Resources Canada

1. Using FME for Topographical Data Generalization at Natural Resources Canada Daniel Pilon Senior project officer at NRCan

2. Overview • Map generalization at NRCan • Principles of good map generalization • ETL generalization patterns

3. Map Generalization The process of reducing detail on a map Very complex: manually or automatically

4. Generalization Paradigm Shift • Manually • Comply to very strict specifications • Prepackage products 250K, 1M, 7.5M • Finality on its own Art Craft Process • Automatically (faster turnaround) • Less complex specifications • On demand scale production • Integrated into other products Commodity Process

5. Generalization Paradigm ShiftExample WMS • Web Mapping Service • 1 : 30 000 000 • 1 : 7 500 000 • 1 : 1 000 000 • 1 : 250 000 • 1 : 50 000

6. Generalization Paradigm ShiftExample WMS 1 : 40 000 000

7. Generalization Paradigm ShiftExample WMS 1 : 5 000 000

8. Generalization Paradigm ShiftExample WMS 1 : 1 000 000

9. Generalization Paradigm ShiftExample WMS Threshold: 1: 120 000 1 : 121 000 (serving 1: 250 000) 1 : 119 000 (serving 1: 50 000)

10. FME for Generalization…Why? • Few generalization solutions • No complete generalization solution • Some European solutions • Very complex software to use… • Tackle European problems

11. FME for Generalization…Why?

12. FME for Generalization…Why? • Few generalization solutions • No complete generalization solution • Some European solutions • Very complex software to use… • Tackle European problems • FME as a generalization tool… • General purpose spatial manipulation engine • Reliable and fast • Easily add extensions (python)

13. Overview Map generalization at NRCan Principles of good map generalization ETL generalization patterns

14. The principles of Good Genralization Automation Generalization cases • Simple and efficient decomposition of the tasks performed by the cartographers • Objective tools to characterize map objects • Tools to edit modify the spatial objects • Reactive control on the map object’s state during the process • Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

15. Generalization Case • It is a formalization of the cartographer’s knowledge (generalization rules) and a communication tool between the cartographer and the software engineer • Generalization cases do not tell us what algorithms or what parameters to use

16. Generalization Case example • For the Vegetation • For all the Vegetation which breaks the constraint Dimension: Minimum area • If the Vegetation breaks the constraint: Position proximity • Aggregate the area • Else • Eliminate the area

17. The principles of Good Genralization Automation Generalization cases • Simple and efficient decomposition of the tasks performed by the cartographers • Objective tools to characterize map objects • Tools to edit modify the spatial objects • Reactive control on the map object’s state during the process • Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

18. Measures • Measures are used: • To calculate the characteristics of map objects (or group of map objects ) • Before the generalization: to know how to generalize • After the generalization: to assess the success of the applied generalization operations

19. Measure Examples • Unary measures Length Area/Perimeter Circularity ratio Oriented Bounding box Triangulation Bounding box

20. Measure Examples • Binary measures Distance Triangulation Spatial relationship

21. Measure Examples H1 S1 H3 S1 H2 S1 • N-ary mesaures H1 S1 H1 S1 H3 S2 H1 S1 H1 S1 H3 S2 H2 S2 H2 S2 H2 S1 H2 S2 H3 S3 H1 S1 H3 S3 Stream ordering (Strahler or Horton)

22. The principles of Good Genralization Automation Generalization cases • Simple and efficient decomposition of the tasks performed by the cartographers • Objective tools to characterize map objects • Tools to edit the spatial objects • Reactive control on the map object’s state during the process • Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

23. Generalization Operators • Generalization operators are typical transformations applied on spatial objects. They allow the decomposition of the generalization process into several sub-problems in order to manage complexity • Generalization operator are implemented using different generalization algorithms. Ex.: For line simplification: Douglas & Peucker, Lang, Sherbend

24. Generalization Operator Examples • Selection • Simplification • Smoothing • Collapse

25. Generalization Operators Examples • Aggregation • Typification • Displacement

26. The principles of Good Genralization Automation Generalization cases • Simple and efficient decomposition of the tasks performed by the cartographers • Objective tools to characterize map objects • Tools to edit modify the spatial objects • Reactive control on the map object’s state during the process • Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

27. Constraints • Constraints are rules applied to data in order to comply with requirements of the target map specifications (ex: An area must respect the minimum size threshold) • Constraint are used: • To trigger a generalization operation • To assess the result of a generalization operation • To achieve generalization, a feature must fulfill several constraints

28. The principles of Good Genralization Automation Generalization cases • Simple and efficient decomposition of the tasks performed by the cartographers • Objective tools to characterize map objects • Tools to edit modify the spatial objects • Reactive control on the map object’s state during the process • Unambiguous rules in order to guide the process Measure Operator Constraint Behaviour pattern

29. Generalization Behaviour • Provides the link between constraints, generalization operator and measures • Implements mechanism for choosing algorithms and parameters • Implements mechanism for choosing alternate scenarios in case of constraint failure

30. Generalization Behaviour Example for Line Simplification Reset to the original geometry Determine an alternate algorithm and/or parameters ETL Generalization Patterns Failed Apply the algorithm on the feature Asses the results through a set of constraints Determine the algorithm to use Success Measures on the feature

31. Overview Map generalization at NRCan Principles of good map generalization ETL generalization patterns

32. ETL Generalization Patterns • General and reusable solution to a commonly occurring problem in generalization when used in an ETL context • Three stages pattern • Meta-algorithm pattern

33. Three Stages Pattern • Used when generalization operator(s) are applied on different part of the same feature • Solution: • Create a pseudo-object representing each generalization operation to be done on the real objects • Evaluate the constraint against the pseudo-objects created in 1 • Apply on the real objects the pseudo-objects that meet the constraint evaluated in 2

34. Three Stages Pattern Example • Example with amalgamator operator • If you implement a one stage process

35. Three Stages Pattern Example • Example with amalgamator operator • If you implement a three stages process • First create the amalgamation zones • Second you remove unwanted zone • Dissolve the area

36. Meta-Algorithm Pattern • Used when it is too complex to program the behaviour and the constraint in an ETL • Solution: • Select the algorithm(s) • Select the constraint(s) to implement • Select a behaviour pattern • Implement the solution in a high level language • Wrap the solution in a transformer

37. Meta-Algorithm Pattern Example • Example with Sherbend • Implementation of the Wang algorithm

38. Meta-Algorithm Pattern Example Line simplification Basic generalization operation We all thought of the Douglas-Peucker algorithm to solve that problem… Does it simulate the work of the cartographer well enough? An example with contours

39. Meta-Algorithm Pattern Example Contour line simplification Red: Original contours Black: Contours with DP 50m

40. Meta-Algorithm Pattern Example Contour line simplification Black: Contours with DP 50m The results are unacceptable Does it simulate the work of the cartographer? No Douglas-Peucker line simplification is not doing a good job of generalizing natural features

41. Meta-Algorithm Pattern Example Contour line simplification Red: Original contours Black: Contours with Sherbend algorithm with a parameter of 150m The Sherbend algorithm will remove the bends below the tolerance and keep the ones above it

42. Meta-Algorithm Pattern Example • Example with Sherbend • Implementation of the Wang algorithm • Implementation of constraints • Self intersection • Line crossing • Sidedness

43. Meta-Algorithm Pattern Example • Implementation of constraints • Self intersection • Line crossing • Sidedness

44. Meta-Algorithm Pattern Example • Example with Sherbend • Implementation of the Wang algorithm • Implementation of constraints • Self intersection • Line crossing • Sidedness • Implementation of a scenario to resolve conflicts (behaviour pattern)

45. Meta-Algorithm Pattern Example • Scenario to resolve conflitcs • Implementation of an iterative process to resolve constraints

46. Meta-Algorithm Pattern Example • Example with Sherbend • Implementation of the Wang algorithm • Implementation of constraints • Self intersection • Line crossing • Sidedness • Implementation of a scenario to resolve conflicts (behaviour pattern) • Creation of a new transformer

47. Meta-Alorithm Pattern • Meta-Algrorithms can be developed by user in FME • FME Python extension (Python caller) • Open source libraries • Shapely for spatial manipulation and spatial relationship (http://pypi.python.org/pypi/Shapely) • RTree for spatial indexing (http://pypi.python.org/pypi/Rtree/) • Already implemented the following meta-algorithms • Douglas Peucker • Convex • Spike • Smoothing • Sherbend

48. Future Works • Develop new generalization patterns applied to ETL • Develop new meta-algorithms • Start the generalization of the hydrographic network

49. Thank You! • Questions? • For more information: • Daniel Pilon • Natural Resources Canada • dpilon@nrcan.gc.ca • WWW.GEOGRATIS.GC.CA • WWW.GEOBASE.CA