1 / 41

Python and Geoprocessing

Python and Geoprocessing. Ryan Branch, GISP, CFM GIS Analyst Melissa Williams, PE, GISP Senior Project Engineer. 2014 Ohio GIS Conference September 22 - 24, 2014 | Hyatt Regency Columbus| Columbus, Ohio. Agenda. 1 Simple Tasks. Joins, Splitting Features, Field Calculation.

cherie
Download Presentation

Python and Geoprocessing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Python and Geoprocessing Ryan Branch, GISP, CFM GIS Analyst Melissa Williams, PE, GISP Senior Project Engineer 2014 Ohio GIS Conference September 22 - 24, 2014 | Hyatt Regency Columbus| Columbus, Ohio

  2. Agenda 1Simple Tasks Joins, Splitting Features, Field Calculation 2Other Tasks Iteration, ArcMap Tools, Model Builder Application 3Interfacing With Non-GIS Data Text, Excel, SQL Server, Access

  3. Simple Tasks Joins, Splitting Features, Field Calculation 1

  4. Table Management • Joins • arcpy.AddJoin_management(Table1, JoinF1, Table2, JoinF2, Condition) • Attributing fields • arcpy.CalculateField_management(Table, TargetField, DesiredInput, InputType)

  5. Table Management Split by Attributes • TableSelect_analysis(Input, Output, Condition)

  6. File Management • Path split • os.path.split (head, tail) • C:\Keep\Geodatabase.gdb\BufferPoints

  7. File Management oneper = "[EVENT_TYP] = '1 PERCENT CHANCE'" fourper = "[EVENT_TYP] = '4 PERCENT CHANCE'" tenper = "[EVENT_TYP] = '10 PERCENT CHANCE'" twopwer = "[EVENT_TYP] = '2 PERCENT CHANCE'" pointtwoper = "[EVENT_TYP] = '0.2 PERCENT CHANCE'“ fc= r"V:\1735\temporary\users\RBranch\KegWeep\Working.gdb\FolderNetwork" for field in arcpy.da.SearchCursor(fc,("NetworkLocFull")): try: f = field[0] arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_1",oneper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_4",fourper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_10",tenper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_2",twopwer) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_02",pointtwoper) This takes the field names within the list “field” and splits off their tail, then telling ArcMap to create a new table whose file name is that of the “field” plus “L_XS_Elev_1” when a specific attribute (namely, “EVENT_TYP”) is “1 PERCENT CHANCE”

  8. Other Tasks Iteration, ArcMap Tools, Model Builder Application 2

  9. Iteration • iglob • Used in conjunction with “*” placeholder • Finds all files/pathnames in a specific set • File path placeholders • “C:\temporary\Hydraulics\*\Hydraulic_Models\Spatial_Files\L_XS_Elev_1.dbf” This would find every table named “L_XS_Elev_1.dbf” within this set of folders

  10. Iteration iglob and variable simplification tab02 = glob.iglob(r“C:\temporary\users\RBranch\KegWeep\Hydraulics\10240001\Hydraulic_Data\*\Hydraulic_Models\Spatial_Files\fema_2012_07_27.mdb\L_XS_Elev_02") tab1 = glob.iglob(r“C:\temporary\users\RBranch\KegWeep\Hydraulics\10240001\Hydraulic_Data\*\Hydraulic_Models\Spatial_Files\fema_2012_07_27.mdb\L_XS_Elev_1") for ret1 in tab02: dir = r"C:\Temp\GridWork\KegWeepingData\XS_Combined.gdb\XS_Combined" arcpy.AddJoin_management(ret1, dir)

  11. ArcMap Tools Creating TINs # Process: Create TIN arcpy.CreateTin_3d(TIN_Name, Coordinate_System, Input_Data, "DELAUNAY")

  12. ArcMap Tools Creating TINs

  13. Model Builder • Code exports • ArcMap can export a tool or model as a .py

  14. # Local variables: Ras03Min = Ground_elevation_raster Ras04SNul = Ras03Min Ras05Int = Ras04SNul Ras06Poly = Ras05Int Ras07Dislv = Ras06Poly Ras08Smooth = Ras07Dislv Ras08Smooth_Pnt = Ras07Dislv Ras01Ras = Raster_sampling_size__units_according_to_coord_system_ Ras02Extct = Ras01Ras Output_File_GDB = OutputLocation TIN = Output_File_GDB # Process: Create File GDB arcpy.CreateFileGDB_management(OutputLocation, "%OutputDatabaseName%", "10.0") # Process: Create TIN arcpy.CreateTin_3d(TIN, Coordinate_System, Backwater_polygon__use_Hard_Replace_option_, "DELAUNAY") # Process: TIN to Raster arcpy.TinRaster_3d(TIN, Ras01Ras, "FLOAT", "LINEAR", Raster_sampling_size__units_according_to_coord_system_, "1")

  15. Interfacing With Non-GIS Data Text, Excel, SQL Server, Access 3

  16. Text Files • Open/create a text file • textFile= open(‘C:\\example.txt', 'r') • Options when opening files:

  17. Text Files • Read an existing text file • Read whole file at once: • textContent = textFile.read() • Read all the lines and return a list: • textList = textFile.readlines() • Read a single line at a time: • textLine= textFile.readline() • Create loop to look through each line: • for eachLine in textFile:

  18. Text Files • Write to a text file • Use write function, \n for new line • textFile.write(“hello world\n”) • Save/close text file • Use whenever done reading or writing • textFile.close()

  19. Text Files • Example:

  20. Excel • Connect to Excel • XlsxWriter(https://xlsxwriter.readthedocs.org/) • - Can only write new files, not read/modify • OpenPyXL(https://pythonhosted.org/openpyxl/) • - For Excel 2007 only • EasyExcel class (http://anselmo.homeunix.net/ebooks/pythonwin32/ch09.htm) • - Only allows one Excel file open at a time

  21. Excel • EasyExcel Class • Useful Functions:

  22. Excel • Problems with EasyExcel • Could not have geoprocessing web service open Excel and write to file • Workaround was to write to a text file, save as CSV, then import into preset Excel template using VBA script

  23. SQL Server/Access • Establish connection: SQL Server • Use pyodbc or other module that allows you to connect to other non-GIS databases • SQL Server must allow external connections, user with read/write permissions

  24. SQL Server/Access • Establish connection: Access • Use pyodbc or other module that allows you to connect to other non-GIS databases

  25. SQL Server/Access • Querying Data • Create cursor after creating connection • cursor = cnxn.cursor() • Use execute and fetch function to perform query • cursor.execute(“select user_id from users”) • rows = cursor.fetchall() • Use row objects to read data one row at a time • for row in rows: • Access data by column index or name • row[1] • row.ColumnName

  26. SQL Server/Access • Example Query:

  27. SQL Server/Access • Use execute to perform changes, commit to save • Insert • Update • row.user_id = newValue • cnxn.commit() • Delete • cursor.execute(“delete from products where id <> ‘test’”) • cnxn.commit()

  28. SQL Server/Access • Problems with connections not closing in geoprocessing web service • Pyodbc module did not close connection • Solution: use pypyodbc or query layers in ArcMap

  29. SQL Server/Access

  30. SQL Server/Access • Problems joining SQL Server table • Join field in SQL table was an integer with “Identity” property (autonumbering) = Yes • ArcGIS viewed field as an ObjectID instead of an integer • Workaround was to use query layers in ArcMap • Use advanced options to set multiple fields as the unique identifier fields

  31. SQL Server/Access Problems joining table in SQL Server

  32. Questions? Inquiries?

  33. Full Process Example • Splitting data out of a master table • Cross-sections for multiple streams each have elevation attributes in wrong format • Combining tables into properly formatted table • In order to process everything quickly, need one feature with each of 5 sets of elevations • Attach & attribute a line file with each split table • Creating TINs out of tables and cross-section layer • Use resulting datasets to create a TIN for each of 5 sets of elevations along each stream

  34. Full Process Example Split master table (with improper field types) into 5 tables, one for each elevation set

  35. Full Process Example Take sets of elevations from grouped tables and splitthem out to individual tables oneper = "[EVENT_TYP] = '1 PERCENT CHANCE'" fourper = "[EVENT_TYP] = '4 PERCENT CHANCE'" tenper = "[EVENT_TYP] = '10 PERCENT CHANCE'" twopwer = "[EVENT_TYP] = '2 PERCENT CHANCE'" pointtwoper = "[EVENT_TYP] = '0.2 PERCENT CHANCE'"

  36. Full Process Example f = field[0] arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_1",oneper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_4",fourper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_10",tenper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_2",twopwer) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_02",pointtwoper) print field[0] Export specific types of elevations into new tables

  37. Full Process Example Split master table (with improper field types) into 5 tables, one for each elevation set oneper = "[EVENT_TYP] = '1 PERCENT CHANCE'" fourper = "[EVENT_TYP] = '4 PERCENT CHANCE'" tenper = "[EVENT_TYP] = '10 PERCENT CHANCE'" twopwer = "[EVENT_TYP] = '2 PERCENT CHANCE'" pointtwoper = "[EVENT_TYP] = '0.2 PERCENT CHANCE'" print "started" fc = r"V:\1735\temporary\users\RBranch\KegWeep\Working.gdb\FolderNetwork" for field in arcpy.da.SearchCursor(fc,("NetworkLocFull")): try: f = field[0] arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_1",oneper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_4",fourper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_10",tenper) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_2",twopwer) arcpy.TableSelect_analysis(f,os.path.split(f)[0] + r"\L_XS_Elev_02",pointtwoper) print field[0] except: print "failed on " + str(field[0]) print "finished"

  38. Full Process Example Find all groups of like tables (each elevation set) tab02 = r"V:\1735\temporary\users\RBranch\KegWeep\Hydraulics\10240001\Hydraulic_Data\*\Hydraulic_Models\Spatial_Files\fema_2012_07_27.mdb\L_XS_Elev_02“ tab1 = r"V:\1735\temporary\users\RBranch\KegWeep\Hydraulics\10240001\Hydraulic_Data\*\Hydraulic_Models\Spatial_Files\fema_2012_07_27.mdb\L_XS_Elev_1“ tab2 = r"V:\1735\temporary\users\RBranch\KegWeep\Hydraulics\10240001\Hydraulic_Data\*\Hydraulic_Models\Spatial_Files\fema_2012_07_27.mdb\L_XS_Elev_2“ tab4 = r"V:\1735\temporary\users\RBranch\KegWeep\Hydraulics\10240001\Hydraulic_Data\*\Hydraulic_Models\Spatial_Files\fema_2012_07_27.mdb\L_XS_Elev_4“ tab10 = r"V:\1735\temporary\users\RBranch\KegWeep\Hydraulics\10240001\Hydraulic_Data\*\Hydraulic_Models\Spatial_Files\fema_2012_07_27.mdb\L_XS_Elev_10"

  39. Full Process Example Join to a combined cross-section feature, add one set of elevations to the feature for ret in tab02: print ret arcpy.AddJoin_management(tab02, C:\\Temp\\GridWork\\KegWeepingData\\XS_Combined.gdb\\XS_Combined) arcpy.CalculateField_management(C:\\Temp\\GridWork\\KegWeepingData\\XS_Combined.gdb\\XS_Combined, XS_Combined.El_02, L_XS_Elev_02.WSEL, "PYTHON_9.3") Repeat for each set of elevations

  40. Full Process Example Create TINs from list of streams and from combined cross-section feature

  41. Full Process Example With one overall TIN, it is now simple enough to create rasters and polygons in a single, coherent dataset

More Related