Smallworld Core Spatial Technology™ and FME: Configuration Tips and Tricks

1. Smallworld Core Spatial Technology™ and FME: Configuration Tips and Tricks Presented by Alfred Sawatzky 2007 Smallworld Users Conference

2. Learning Objectives • Easily install the FME Application for Smallworld Core Spatial Technology™ • Understand the unique-to-Smallworld Core Spatial Technology™ coordinate system configurations for FME • Find out how to automate data extracts from Smallworld Core Spatial Technology™ via FME using batch files • Safe Software has workshops to show you how to use their tools. This presentation will show you how to implement the Smallworld Core Spatial Technology™/FME interface based on real world lessons learned.

3. What is FME? • Feature Manipulation Engine sold by Safe Software • Allows users or processes to… “Move data from multiple sources, reformat and/or cleanse it, and load it into other database(s), a data mart, a data warehouse, or other operational systems for analysis” • Core Capabilities • Translate • Transform • Federate • Distribute

4. Translate Translate data from/to over 190 CAD, GIS, vector, raster and database formats OGC Smallworld Core Spatial Technology™ MapInfo Google Autodesk GeoRSS Intergraph ESRI

5. Transform Data restructuring both in terms of attribution and geometry is key!

6. Federate Let users access data where it is and share as federated (unified) data view – no need to copy to a data store or put into a unified schema

7. Distribute Distribute data to users in the structure and format users desire – and via the web technology they prefer

8. FME and Smallworld Core Spatial Technology™ : Examples • Visualization: Using FME as a Spatial Business Intelligence Visualization tool • FME provides a nice visual tool (Workbench) to allow GIS data analysts to process spatial data without knowing Smallworld Core Spatial Technology™/Smallworld Magik™. • Examples: • http://sworldwatch.blogspot.com/2007/01/fme-as-spatial-intelligence-tool-for.html • http://sworldwatch.blogspot.com/2007/04/using-fme-to-report-lines-lengths-per.html • Validation: Using FME to validate Smallworld Core Spatial Technology™ data • Migration: Migrating from legacy system into Smallworld Core Spatial Technology™ • Dissemination: Distributing Smallworld Core Spatial Technology™ based data into other formats.

9. How do FME and Smallworld Core Spatial Technology™ Interface? • Need a FME installation and license from Safe Software. • for Core Spatial Technology™ 4.x need a minimum of FME 2003 ICE and a maximum of FME 2007

10. How do FME and Smallworld Core Spatial Technology™ Interface? • Need some Smallworld Magik™ code that interfaces Smallworld Core Spatial Technology™ with FME. Licensed depending on which vendor you purchase from. • GE Energy • 4.1 Translation Suite or FME 4.0. Earlier versions of FME on Smallworld Core Spatial Technology™ (for Core Spatial Technology™ 3.x) also available but they do not work with FME Workbench. • Spatial Business Solutions • FME Plugin • http://www.spatialbiz.com/pages/fme_plugin_spec.htm

11. Installing FME • This presentation will focus on installing the interface provided by GE Energy in the Core Spatial Technology™ 4.1 SWAF environment. • Differences between 4.0 and 4.1 will be noted. • Download and install FME 2007 from http://www.safe.com. • Be sure to specify the “Smallworld Core Spatial Technology™ Edition” when downloading the installer and requesting licenses. • There are a number of licensing options available to you.

12. Installing FME-Smallworld Core Spatial Technology™ Interface(sworldswaf.dll) • Be sure you are using the correct version of sworldswaf.dll in your FME_HOME directory (eg., c:\program files\fme) • Smallworld Core Spatial Technology™ 4.0 FME Interface requires dll version 1.0.1.0 (this is the one that currently ships with FME 2006 GB) • Smallworld Core Spatial Technology™ 4.1 FME Interface requires dll version 1.0.0.11 (this is the one that currently ships with FME 2007) • Contact your GE Energy Help Desk for the correct DLL. • See http://sworldwatch.blogspot.com/2007/07/fme-41-translator-dll-problem.htmlfor more details until GE Energy publishes a TSB for this issue.

13. Installing FME-Smallworld Core Spatial Technology™ Interface • The Smallworld Core Spatial Technology™ on FME interface is delivered as a Smallworld Magik™ product. • Typically install the product (fme410) in a folder next to the sw410 product folder (eg., c:\sw410\fme410) • You will need to make sure that the fme410\product.def file is known to your image (eg., add the full path to the SW_PRODUCTS_PATH environment variable)

14. Installing FME-Smallworld Core Spatial Technology™ Interface • Modify your image build module or some other suitable module to “require” these FME modules: • fme_plugin • fme_import_plugin • fme_export_plugin

15. Installing FME-Smallworld Core Spatial Technology™ Interface(SWAF FME Application) • The FME 4.1 documentation instructs you to load the swaf_fme_application module. sw_module_manager.load_module(:swaf_fme_application) • While this is useful for a quick demo it is not very helpful for integrating into your existing applications. • Customers say: “How do we integrate the FME functionality into our existing applications?”

16. Installing FME-Smallworld Core Spatial Technology™ Interface • How do I integrate the FME interface into my existing application(s)? • module.def • Modify the module.def for your application(s) to “require” • fme_plugin • fme_import_plugin • fme_export_plugin

17. Installing FME-Smallworld Core Spatial Technology™ Interface • How do I integrate the FME interface into my existing application(s)? • config.xml • Modify <application path>\resources\base\data\config.xml and add the following plugin definition to the list of plugins. <plugin name="fme" class_name="fme_plugin"> <properties> <property name="port_number" value="30000"/> </properties> </plugin>

18. Installing FME-Smallworld Core Spatial Technology™ Interface • How do I integrate the FME interface into my existing application(s)? • explorer_model_config.xml • Modify a “suitable” (see next slide) resources\base\data\explorer_model_config.xml to include the following two entries: <plugin name="fme_export" class_name="explorer_model_fme_export_plugin" /> <plugin name="fme_import" class_name="explorer_model_fme_import_plugin" />

19. Installing FME-Smallworld Core Spatial Technology™ Interface • How do I integrate the FME interface into my existing application(s)? • explorer_model_config.xml • “Suitable” depends on what the model_config_definition_* property settings are for your explorer plugin in the application’s resources\base\data\config.xml file. You can choose to modify the Explorer for only a single application or for many applications. <plugin name="explorer" class_name="explorer_plugin"> <properties> <property name="model_config_definition_file_name“ value="explorer_model_config.xml"/> <property name="model_config_definition_module_name" value="swaf_fme_application"/> </properties> </plugin>

20. Installing FME-Smallworld Core Spatial Technology™ Interface • Restart your application after you have made all the XML changes. • Start up the Explorer tool. You should see two new FME folders at the top of the left pane.

21. Pull not Push • First set up FME/Smallworld Core Spatial Technology™ interface as a data source to serve up certain tables/records on request. Hint For large “My Datasets” lists, the Click and Drag does not work as expected. Use Search Restrictions to Help you out. Click and Drag

22. Pull not Push • Start FME tools to request data from the FME/Smallworld Core Spatial Technology™ data source

23. Pull not Push

24. Pull not Push • We have used FME Universal Viewer to show the connection between FME and Smallworld Core Spatial Technology™. • Once the connection is established we can just as easily use FME Workbench or FME Universal Translator to connect to Smallworld Core Spatial Technology™.

25. FME TICS and what that means for you • At application startup, host port is assigned to FME TICS server (default 30000). • A port on a machine can be used by only one service. • If your clients are running on Citrix or Terminal Server this will cause a problem because you have many Smallworld Core Spatial Technology™/FME users using one machine. Only the first user to log in will be bound to a port. • Unless… • You are using FME for Smallworld Core Spatial Technology™ 4.1; or • You have some kind of customization for FME for Smallworld Core Spatial Technology™ 4.0 that attempts to find an unused port for the FME TICS server. • E.g., p4_1rpc.Smallworld Magik™

26. FME TICS and what that means for you • If you have a patch that allows multiple FME on Smallworld Core Spatial Technology™ sessions to use different ports on Citrix or Terminal Server, you are confronted with another issue. • How can a Smallworld Core Spatial Technology™ user know which port to connect their FME tools to? Currently, the only way to do that is to inspect the command window for the current Smallworld Core Spatial Technology™ image and look for something like… TICS server started FME TICS Server Running on 30000 • Not very user-friendly, but…

27. A few simple fixes can make it easier(part 1) _method explorer_model_fme_import_plugin.setup_explorer_elements() ## ## Sets up the explorer elements for this plugin. ## _if .explorer_element _is _unset _then # START CHANGE # (2007-03-16 Alfred Sawatzky, RPC Inc.) : #_local group_name << _self.message(:group_name) _local server << fme_tics_server.current_server _local group_name << _self.message(:group_name, server.port_number, system.host_name, server.application_coordinate_system.external_cs_name) # # END CHANGE # Create the top level group element for this plugin .explorer_element << explorer_group_element.new_with( :owner, _self, :name, group_name, :image, { :explorer_fme, :fme_engine }, :column_headings, { _self.message(:name_heading) }, :visible_field_names, { :name }, :column_widths, { 400 } ) _endif _endmethod $ read_message_patch(:explorer_model_fme_import_plugin,_unset,:en_us) $ :group_name Import from FME (Port #1 on #2. Coordinate System '#3') $

28. A few simple fixes can make it easier(part 2) _method explorer_model_fme_export_plugin.setup_explorer_elements() ## ## Sets up the explorer elements for this plugin. ## _if .explorer_element _is _unset _then # START CHANGE # (2007-03-16 Alfred Sawatzky, RPC Inc.) : #_local group_name << _self.message(:group_name) _local server << fme_tics_server.current_server _local group_name << _self.message(:group_name, server.port_number, system.host_name, server.application_coordinate_system.external_cs_name) # # END CHANGE # Create the top level group element for this plugin .explorer_element << explorer_group_element.new_with( :owner, _self, :name, group_name, :image, { :explorer_fme, :fme_engine }, :column_headings, { _self.message(:name_heading) }, :visible_field_names, { :name }, :column_widths, { 400 } ) _endif _endmethod $ read_message_patch(:explorer_model_fme_export_plugin,_unset,:en_us) $ :group_name Export to FME (Port #1 on #2. Coordinate System '#3') $

29. Start FME Universal Translator • For this example we will use the FME Universal Translator for a quick translation. • Lessons learned here about host, port and coordinate system are transferable to the FME Workbench and FME Universal Viewer.

30. Start FME Universal Translator ?

31. Coordinate Systems • FME on Smallworld Core Spatial Technology™ interface uses the Application Coordinate System • not the dataset, view or trail coordinate systems. • This is an important distinction to remember!! • What if I can't find my Smallworld Core Spatial Technology™ Coordinate System in FME! • It is not important that the Smallworld Core Spatial Technology™ and FME coordinate systems have the exact same name. • It is important that they have the same coordinate system definition. • It is very likely that the application coordinate system that your Smallworld Core Spatial Technology™ application is using will not be defined in the FME coordinate system gallery.

32. Comparing FME and Smallworld Core Spatial Technology™ Coordinate Systems • Start Smallworld Core Spatial Technology™ Coordinate System Editor

33. Comparing FME and Smallworld Core Spatial Technology™ Coordinate Systems • Show Properties of the selected FME Coordinate System

34. Comparing FME and Smallworld Core Spatial Technology™ Coordinate Systems • Compare the two definitions to see if they are the same. • For help with FME coordinate system parameters see http://www.safe.com/support/online-documentation/CoordSys/coordinate_system_help.htm

35. Defining a New FME Coordinate System • Look at http://www.safe.com/support/online-documentation/CoordSys/coordinate_system_help.htm#adding_a_custom_coordinate_system.htm • Edit %FME_HOME%/reproject/MyCoordSysDefs.fme on your FME client. • When you update to a newer version of FME, the MyCoordSysDefs.fme file will not be overwritten. • If you are working with custom mapping files and you set coordinate system definitions, remember to remove the definitions once they are defined in the MyCoordSysDefs.fme file – the same coordinate system cannot be defined in both places.

36. Smallworld Core Spatial Technology™/FME Coordinate System Problem! • In some instances (eg., Cambridge DB), the application coordinate system is Lat-Long with units = mm. • Smallworld Core Spatial Technology™ automatically assigns a Cartesian unit to an angular coordinate system for the application CS. • FME does not support Cartesian units for its LL projection. • The only out-of-the-box way around this problem is to modify the :units on the Smallworld Core Spatial Technology™ Application definition to be :degrees. • But this can have other impacts (e.g., snapping tolerances)

37. Smallworld Core Spatial Technology™/FME Coordinate System Solution! • It is only a matter of a little Smallworld Magik™  • Add a new property to the fme_plugin class called :coordinate_system_name • Modify fme_plugin.application_coordinate_system • Add new property to config.xml with a coordinate system name that you know FME supports.

38. Smallworld Core Spatial Technology™/FME Coordinate System Solution (Smallworld Magik™)! _pragma(classify_level=advanced) ## ## The name of the 'application_coordinate_system' to use when ## starting the fme_tics_server. If no name is given or the ## name is not valid, then the fme_tics_server will use the ## coordinate_system of the current application. ## fme_plugin.def_property( :coordinate_system_name, :type, :string, :default_value, "" ) $ _pragma(classify_level=restricted) _method fme_plugin.application_coordinate_system ## ## Returns the current application coordinate system ## >> _if ( app << _self.application ) _isnt _unset _then # START CHANGE # (2007-08-16 Alfred Sawatzky, RPC Inc.) : # _local db << app.database _local cs << app.database.coordinate_system(_self.coordinate_system_name) >> cs.default(app.coordinate_system) #>> app.coordinate_system # # END CHANGE _endif _endmethod $

39. Smallworld Core Spatial Technology™/FME Coordinate System Solution (XML)! • Modify <application path>\resources\base\data\config.xml and add the new plugin property. <plugin name="fme" class_name="fme_plugin"> <properties> <property name="port_number" value="30000"/> <property name="coordinate_system_name" value="world_longlat_wgs84_degree"/> </properties> </plugin> Matches the FME LL Coordinate System

40. Creating Batch Processes • Many customers want to create batch processes that interface Smallworld Core Spatial Technology™ with FME. • First create and test a .fmw (or .fme) file to extract data from Smallworld Core Spatial Technology™. • Then write some Smallworld Magik™ code to launch FME (command line) with your .fmw (or .fme) file.

41. Creating Batch Processes (Smallworld Magik™) _block _global fme_tics_server _local current_server # start the fme_tics_server if not already running _if (current_server << fme_tics_server.current_server) _is _unset _orif _not current_server.acp_running? _then fme_tics_server.start(_unset,_unset,:gis) current_server << fme_tics_server.current_server _endif # indicate which records to prepare for export _local export_records << rwo_set.new_from(gis_program_manager.cached_dataset(:gis).collections[:min_road]) current_server.set_output_rwo_set(export_records) # run the command line FME _local command << {"fme","c:\temp\sworld2kml.fmw"} system.do_command(command) _endblock $

42. Creating Batch Processes (Smallworld Magik™) • If you created a .fmw file that had “Published Parameters”…

43. Creating Batch Processes (Smallworld Magik™) • … you could write something like… _block _global fme_tics_server _local current_server # start the fme_tics_server if not already running _if (current_server << fme_tics_server.current_server) _is _unset _orif _not current_server.acp_running? _then fme_tics_server.start(_unset,_unset,:gis) current_server << fme_tics_server.current_server _endif _local export_records << rwo_set.new_from(gis_program_manager.cached_dataset(:gis).collections[:min_road]) current_server.set_output_rwo_set(export_records) _local command << {"fme","c:\temp\sworld2kml.fmw", "--DestDataset_KML",'"c:\temp\cambridge_roads.kml"', "--PORT_SWORLDSWAF",current_server.port_number.write_string, "--COORDSYS","LL"} system.do_command(command) _endblock $

44. Creating Batch Processes (Smallworld Magik™) • Now that you have the Smallworld Magik™ batch steps, you could further wrap that in a .cmd or .bat file that: • Starts Smallworld Core Spatial Technology™ • Runs your FME batch script • Closes Smallworld Core Spatial Technology™

45. FME on Smallworld Core Spatial Technology™ User Extensions • The interface between Smallworld Core Spatial Technology™ and FME is very powerful once it has been configured correctly. • I have seen customers create: • Batch FME output • New GUIs that use the FME Engine under the hood • Enhance the Smallworld Core Spatial Technology™ Explorer “Export to…” functionality to include “Export to FME…”

46. FME on Smallworld Core Spatial Technology™ User Extensions

47. Assistance • http://www.safe.com/support/online-documentation/index.php • http://www.fmepedia.com • an information commons provided by Safe Software Inc. for the FME User Community. • http://groups.google.com/group/fmetalk • http://finance.groups.yahoo.com/group/fme/ • No more posts allowed as of August 16, 2007 but is still available for you to search archives. • support@safe.com

48. Questions? • alfred@redplanetconsulting.com • http://sworldwatch.blogspot.com