GIS Data: Processing ArcToolbox, Geoprocessing and ModelBuilder Spring 2008

1. GIS Data: Processing ArcToolbox, Geoprocessing and ModelBuilderSpring 2008 GISC 6382 Applied GIS UT-Dallas Ron Briggs

2. ArcToolbox, Geoprocessing, ModelBuilder :Overview ArcToolbox provides access to geoprocessing • ArcToolbox is a dockable window in ArcCatalog or ArcMap • Comprises a set of Tools (750+) • Tools support all data types (coverages, shapefiles, gdb, raster) • Multi- step workflow available via Modelbuilder and/or scripting • Command line interface also available for traditionalists and/or power users GISC 6382 Applied GIS UT-Dallas Ron Briggs

3. CAD GDB Database What is geoprocessing? • the modification and analysis of spatial (geographic) data. Projections Conversion Data management Spatial analysis GISC 6382 Applied GIS UT-Dallas Ron Briggs

4. What is Geoprocessing used for? • Data processing/ data automation: Creation, Preparation, Conversion, Management • Data analysis/spatial modeling: Suitability, Sensitivity, Capacity, Risk GISC 6382 Applied GIS UT-Dallas Ron Briggs

5. Accessing geoprocessing tools • Geoprocessing is carried out using tools stored in ArcToolbox • A dockable window available in ArcMap, ArcCatalog, ArcGlobe, or ArcScene • Tools run processes Opens ArcToolbox GISC 6382 Applied GIS UT-Dallas Ron Briggs

6. Inside of ArcToolbox Toolbox • Toolbox: Containerfor toolsets and tools • Note: cannot have a toolbox within a toolbox • Toolset: Logicalcontainer of toolsand other toolsets(i.e. folder) • Tool: Singlegeoprocessingoperation (includesdialogs, models,and scripts) Toolset Tools GISC 6382 Applied GIS UT-Dallas Ron Briggs

7. Coverage Geodatabase Shapefile Tables CAD Layer file Raster Behavior Geoprocessing Supported data types • The tools work with many data types GISC 6382 Applied GIS UT-Dallas Ron Briggs

8. The tools work with many data types …Caution • Often there are different tools for different data types. Often there is one: • For shapefiles and geodatabse features classes • Another for coverages • Another for raster datasets • And they have the same name! • For example, there is a separate Clip tool for each of these input data types • Similarly, there is a separate Define tool for each of these input data types • Occasionally, there is a capability only available for coverages • e.g. conversion of SDTS (Spatial Data Transfer Format) or VPF (a vector format used by military) • Convert first to coverage, then convert coverage to shapefile GISC 6382 Applied GIS UT-Dallas Ron Briggs

9. Tools with the same name • Some tools have the same name • Must distinguish between these tools in scripts • Be sure to get the correct tool for the data type you are processing Clips a coverage Clips a feature class Clips a raster GISC 6382 Applied GIS UT-Dallas Ron Briggs

10. A special note on the Coverages Tools toolbox • Only available if you have ArcInfo Workstation installed • The toolset titles (Analysis, Conversion, Data Management) duplicate many of the toolboxes in ArcToolbox as a whole • You need to use these tools to process coverages • In some cases, there are capabilities here not available for other data types • e.g. conversion of SDTS (Spatial Data Transfer Format) or VPF (a vector format used by military) • Convert first to coverage, then convert coverage to shapefile GISC 6382 Applied GIS UT-Dallas Ron Briggs

11. System (Esri provided) toolboxes • All tools are organized into toolboxes • System toolboxesare provided by ESRI. Number depends on • License level (ArcVIEW, ArcEditor, ArcInfo) • Extensions licensed • and you can create your own toolboxes (and tools) GISC 6382 Applied GIS UT-Dallas Ron Briggs

12. Tool Counts—nearly 750 now (9.2) • Counts vary depending on license/extensions • ArcView: 102 tools • ArcEditor: 104 tools • ArcInfo: 251 tools • Spatial Analyst extension: 158 tools • 3D Analyst extension: 45 tools • Geostatistical Analyst extension: 1 tool • A total of 455 tools came with 9.0 • Feature class tools – 120 • Spatial Analyst tools – 185 • Geocoding tools – 5 • 3D Analyst tools– 20 • Linear Referencing–10 • Coverage tools – 57 Counts as of 9.0. 9.1 has an additional 171 tools 9.2 has additional 121 tools GISC 6382 Applied GIS UT-Dallas Ron Briggs

13. The geoprocessing framework: four ways to run tools 2 Command line 1 3 Models Dialog 4 Scripts 4 ArcObjects GISC 6382 Applied GIS UT-Dallas Ron Briggs

14. Selecting the way to run tools: Choosing between the four ways Selection depends on a user’s needs and abilities • As a dialog inside of any ArcGIS application • Simple, one at a time processing (GIS for dummies?) • Good for ad hoc processing • As a command from a command line • Very efficient for command line afficianados • Supports intelligent auto-completion of commands • As a process in a model (Modelbuilder) • An “executable flow chart” for GIS processing • Powerful multi-step processing, yet simple to create and use • As a function call in a script or program • Similar to AML in ArcInfo 7, but avoids learning proprietary language • provides decision making and looping to support batch processing In this course we will use #1 and #3 GISC 6382 Applied GIS UT-Dallas Ron Briggs

15. Parameters • all tools have one or more parameters • pieces of information that the tool needs in order to run • some parameters are required, • such as the name of the input data set • some parameters are optional • most parameters have a default set, which • you should always review, and • change if it’s inappropriate for your application GISC 6382 Applied GIS UT-Dallas Ron Briggs

16. Environment settings • “Customizable defaults” • Common parameters that are applied to all tools within a geoprocessing session • Settings work (“persist”) in all environments (dialogs, command line, models, scripts,) General settings Current workspace Output coordinate system Raster settings Cell size Mask Geodatabase settings XY Domain M and Z Domains Geodatabase raster Statistics Compression Coverage environment Comparison between prj files Precision for new coverages GISC 6382 Applied GIS UT-Dallas Ron Briggs

17. Four levels of environment settings • User Profile • The settings will persist throughout all applications loaded by the user • Application Level • The settings will apply to all tools executed from that application • For a specific model or script • Settings apply to all tools within this specific model or script. • For an individual tool • Settings apply to a specific tool in a model or script Lower order (e.g. individual tool) settings override higher order (e.g. user profile) settings GISC 6382 Applied GIS UT-Dallas Ron Briggs

18. Using Modelbuilder ArcGIS 9.2 has some major enhancements to Modelbuilder: Whats_New_in_Arctoolbox.pdf GISC 6382 Applied GIS UT-Dallas Ron Briggs

19. Why Use Modelbuilder? • document the processing steps (workflow) that you carried out • You will forget • You may need to do it again • re-run the analysis to experiment with different parameters to see how results are affected • rerun the analysis if you find a mistake • carry out repetitive, multi-step tasks • provide other users (e.g. operations staff) with an exact workflow that they can replicate GISC 6382 Applied GIS UT-Dallas Ron Briggs

20. The Elements of ModelBuilder • Model diagram window • Input variable • Tool • Derived data variable • Toolbar • Add Data or Tools • Layout • Zoom and Pan • Add connection • Run GISC 6382 Applied GIS UT-Dallas Ron Briggs

21. Building a model: constructing • Create a new model • Drag tool into model • Drag data into model or onto a tool • Link data and tool GISC 6382 Applied GIS UT-Dallas Ron Briggs

22. Building a model: running and documenting • Save and rename the model • Run model • Document the model GISC 6382 Applied GIS UT-Dallas Ron Briggs

23. Setting model parameters Or double click on the tool’s process box to open its parameter dialog GISC 6382 Applied GIS UT-Dallas Ron Briggs

24. Element States • Not ready to run (parameters not set) • Ready to run • Has been run (note the grey shadow) GISC 6382 Applied GIS UT-Dallas Ron Briggs

25. An example,which we will now build GISC 6382 Applied GIS UT-Dallas Ron Briggs

26. Alternatives to Modelbuilder:Options for Developing Additional Capabilities • Modelbuilder • Multi-step, sequential processing of ArcTools, using a visual development environment which comes as part of ArcGIS • Python, Jscript and VBScript • standardized and relatively simple scripting languages for repetitive processing, including loops and decision trees, using ArcTools • Python scripts can be generated from Modelbuilder • Visual Basic for Applications • Permits writing of VB macros for sophisticated customization and development within standard ArcGIS (ArcMap/ArcCatalog) interface (and thus requires license for these) • may incorporate ArcObjects, the COM compliant software objects out of which ArcGIS is constructed. • ArcGIS Engine • Set of embeddable GIS components (ArcObjects software objects) for use in building custom applications, independent of ArcGIS interface • Runs under Windows, Unix and Linux, with support for Java, C++, COM and .NET GISC 6382 Applied GIS UT-Dallas Ron Briggs

27. Appendix ESRI GIS Software GISC 6382 Applied GIS UT-Dallas Ron Briggs

28. Environmental Systems Research Institute, Inc. (ESRI) Redlands, CA • Privately held by Jack & Laura Dangermond • One of the 50 largest software companies in the world • Leader in GIS with at least 1/3rd of market • 1 million users (2002) and 2,600 employees • Originator of commercial GIS in 1981 with release of ArcInfo • Released their first GUI (graphics user interface) product, ArcView, in 1991 using proprietary Avenue programming language (for MS Windows, UNIX, Macintosh) • These two product lines (ArcView 3 and ArcInfo 7) combined together in ArcGIS v.8 released in 2000 • complete rewrite based on Microsoft COM/Active X software objects • Only run on MS Windows • completely different interface from earlier ArcView and ArcInfo • old, Avenue based, ArcView still available as ArcView 3.3, • But Old Avenue scripts will not run in ArcGIS 8 • old command line ArcInfo 7 still available as ArcGIS Workstation • Old AML scripts will run in ArcGIS 9 • ArcGIS 9 released in 2004 • Current 2008 release is ArcGIS 9.2 GISC 6382 Applied GIS UT-Dallas Ron Briggs

29. ESRI Product Line-up: ArcGISclient products ArcReader (“adobe acrobat” for maps) & ArcExplorer (spatial data viewer) • Free viewers for geographic data. ArcGIS 9.x Desktop: two primary modules (MS NT/2000/XP only) • ArcMap: for data display, map production, spatial analysis, data editing • ArcCatalog: for data management and preview ArcToolbox, for specialized data conversions and analyses, available as a window in both Available capabilities within these modules are “tiered” • ArcView: viewing, map production, spatial analysis, basic editing • ArcEditor: ArcView, plus specialized editing • ArcInfo: ArcView & ArcEditor plus special analyses and conversions Extensions: for special apps.: Spatial Analyst, 3D Analyst, Geostatistics, Business Analyst, Network Analyst. ArcObjects: build specialized capabilities within ArcMap or ArcCatalog using VB for Applications ArcGIS Workstation (for UNIX and MS NT/2000/XP) • the old command line ArcInfo 7.1 ArcGIS Engine (MS NT/2000/XP) • Set of embeddable GIS components (ArcObjects software objects) for use in building custom applications • Runs under Windows, Unix and Linux, with support for Java, C++, COM and .NET • Replaces MapObjects which were based upon a previous generation of GIS objects Notes: ArcGIS 8 released 2000 to integrate two previous standalone products: ArcView and ArcInfo ArcGIS 9 released 2004 providing the full capability that should have been in ArcGIS 8!!! --full support for all data types (coverages, shapefiles, geodatabases) --full support for all previous geoprocessing analyses --Modelbuilder for scripting and repetitive processing --ArcEngine for building custom applications ArcView 3.3 (the predecessor to ArcGIS 8.x) the only GUI option for UNIX.

30. ESRI Product Line-up: ArcGISserver products(Spring 2007) ArcGIS Server • Permits the creation of server-based specialized GIS applications • Provides full range of GIS capabilities to a user without a desktop GIS ArcIMS (Internet Map Server) • Software to develop Internet server-based mapping and basic analysis • Provides maps and simple query to the user without a desktop GIS SDE (Spatial Database Engine) • middleware to support spatial data storage in standard DBMS • Supports all major industry databases: • Oracle, SQL-Server, IBM DB2, Ingres Note: SDE and ArcIMS now distributed as part of ArcGIS Server package as of 9.2 ArcGIS Services • Server based applications built and operated by ESRI or its partners and made available on the Internet for subscription • Normally charged on a “per transaction” basis, but can be flat fee • presumably built using ArcGIS Server GISC 6382 Applied GIS UT-Dallas Ron Briggs

31. Other ESRI Products: • ArcPAD • Mapping on PDAs (“handhelds”) with Windows CE operating system • ArcLogistics Route • Specialized business application for delivery routing • ArcFM • water and telecom: industry specific facilities management • ArcGIS Extensions • Spatial Analyst: raster data analysis • 3D Analyst: 3-dimensional data display • Geostatistics: surface analysis • Business Analyst: marketing and site selection • Survey Analyst: update of ArcInfo COGO (coordinate geometry) module • Network Analyst: network routing algorithims; shortest path, etc. • Maplex: automated, high quality labeling for maps (now part of ArcGIS 9) • Publisher: creates .MXP maps for reading with ArcReader Extensions work irrespective of ArcView/ArcEditor/ArcInfo tier • BusinessMap: • $99 standalone business mapping (originally Richardson-based MapLynx) GISC 6382 Applied GIS UT-Dallas Ron Briggs

32. ArcInfo ArcEditor ArcView ArcExplorer Browser Internet ArcGIS System Consistent interface Increasing capability Clients c:\ ArcGIS Workstation $ ArcMap ArcCatalog ArcToolbox ArcMap ArcCatalog ArcToolbox ArcMap ArcCatalog ArcToolbox ArcServer Services ArcEngine/ ArcObjects Application Development & Customization ArcIMS Services ArcSDE Services ArcPad Databases Multi-user Geodatabases (in Oracle, SQL Server, IBM DBII, etc) Files (Personal Geodatabase, Shapefiles, Coverages, Grids, tins, etc) Source: ESRI with mods. Handheld/Wireless

33. Some additional detail… GISC 6382 Applied GIS UT-Dallas Ron Briggs

34. Discontinued Products • ArcCAD • CAD product from ESRI • PC ArcInfo • 1st effort at PC based GIS • DOS based, command-line driven • Data not compatible with ArcInfo UNIX • Replaced by ArcInfo 8 and ArcView 3.2 • DAK (Data Automation Kit) • Subset of PC ArcInfo for data preparation for ArcView 3.2 • Atlas/GIS • once a leader in PC-based mapping • Bought by ESRI in 1996 & discontinued in 2001 GISC 6382 Applied GIS UT-Dallas Ron Briggs

35. ArcGIS Version 8/9 With Version 8 & 9, now have two flavors: Desktop: • Largest Microsoft COM/ActiveX application to date • Full GUI interface • Customization via Visual Basic for Applications • New data base concepts: Geodatabase • Runs on XP/2000/NT only • no UNIX version available Workstation: • classic, command-line ArcInfo with AMLs (Arc Macro Language) for customization • same as version 7 and earlier, with minor enhancements • the only option for UNIX, but also available on MS XP • With release of ArcGIS 9, little reason to use unless under UNIX • Now, all capabilities have been moved to ArcGIS GISC 6382 Applied GIS UT-Dallas Ron Briggs

36. ArcGIS Desktop Primary Characteristics • GUI-based tools • ArcCatalog, ArcMap, ArcToolbox • Geodatabase spatial data model supported in standard database environments • MS Access (.msb) for personal applications • File-based geodatabase with 9.2 • Any industry db via SDE for multi-user applications • Modeling of real world as intelligent objects • Houses, poles, not points, lines, polygons • COM/ActiveX components (ArcEngine) for embedding geography in other applications ArcInfo7: simple data complex applications ArcInfo8/9: intelligent data simpler applications GISC 6382 Applied GIS UT-Dallas Ron Briggs

37. ArcGIS 9 Desktop Modules ArcCatalog (schema editor, with VISIO generation) • The base application for ArcInfo Desktop • Windows Explorer-like interface • for organizing access to data and metadata • For launching other Desktop apps: MAP and TOOLBOX ArcMap (object editor) • GUI for map creation and spatial data editing • ArcPlot/ArcEdit (from ArcInfo v. 7) & ArcView 3.2 View/Layout combined • Map projections on the fly (not via conversion as in AV) ArcToolbox (geoprocessor) • An interface to geoprocessing tools • In ArcGIS 8 it was a separate module • In ArcGIS Release 9 it’s an integrated window in ArcCatalog or ArcMap GISC 6382 Applied GIS UT-Dallas Ron Briggs

38. ArcGIS Desktop Capability Tiers: Each tier has the same interfaces (ArcCatalog, ArcMap, and ArcToolbox), but an increasing set of capabilities are available within them (and $ price rises accordingly!) ArcView: • viewing, map production, spatial analysis, basic editing ArcEditor: • ArcView, plus topologic editing of geodatabases and editing of coverages ArcInfo: • ArcView and ArcEditor, plus more geoprocessing analysis, conversions, and full support for coverages. • Old, command line ArcInfo including AML support GISC 6382 Applied GIS UT-Dallas Ron Briggs

39. ArcGIS 9.0 versus ArcGIS 8.3 • Main differences from 8.x are in ArcToolbox • ArcToolbox built into ArcCatalog and ArcMap rather than a separate module • All ArcToolbox tools support all data types (geodatabase, shapefiles, coverages) • 8.3 primarily support coverages • ModelBuilder diagrammatic modeling tool • Invaluable for tracking and replicating geoprocessing steps • New scripting capability for repetitive actions • Python, JScript and VBScript--simpler to use than VB for Applications, the only alternative in 8.x • Old aml (arc macro language from ArcInfo 7) also supported What ArcGIS 8 should have been when it was first released! Incorporates just about everything from ArcInfo 7/ArcView 3. GISC 6382 Applied GIS UT-Dallas Ron Briggs

40. ArcGIS 9.1 & 9.2 versus ArcGIS 9.0 For Spring 2007 we will use ArcGIS 9.2 Ormsby text contains copy of 9.0 Gorr and Kurland contains copy of 9.1 Main differences in 9.1 from 9.0 • General performance improvement and user enhancements (e.g. pause drawing) • Additional geoprocessing tools (63 in ArcView, 86 in ArcEditor, 22 in ArcInfo) • Improved labeling of features on maps (Big change here was in 9.0) • Support for linear referencing • Network Analyst extension for transportation networks (shortest path, etc..) Main differences in 9.2 from 9.1 • Geodatabase and map documents (.mxd files) incompatible! • File based geodatabase • Faster, larger capacity (x 1,000) than MS Access-based personal geodatabase • Higher precision of coordinate storage • Identity/info. and measurement tools rewritten and greatly enhanced • Graphing re-written • Supports brushing (linking) between graph & map (as in Anselin’s GeoDA) software) • Supports cartographic representation • Can re-position features on map without changing coordinate values in geodatabase

41. ComputingEvolution And it will all keep changing! Pervasive Computing Internet Desktop • Small Hardware (Nano) • Wireless Internet • Interoperable • Embedded Workstation Mini Source: ESRI, Inc. Mainframe GISC 6382 Applied GIS UT-Dallas Ron Briggs