1 / 21

Automation Using Python for WVDOT GIS Data Management

Automation Using Python for WVDOT GIS Data Management. Yueming Wu , Ph.D., GISP GIS Manager May 9, 2012. Agenda. Introduction WVDOT Geospatial Data Repository Automation Implementation Key Issues & Solutions Future Work Questions & Answers. Introduction. History.

cormac
Download Presentation

Automation Using Python for WVDOT GIS Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Automation Using Python for WVDOT GIS Data Management Yueming Wu, Ph.D., GISP GIS Manager May 9, 2012

  2. Agenda • Introduction • WVDOT Geospatial Data Repository • Automation Implementation • Key Issues & Solutions • Future Work • Questions & Answers

  3. Introduction

  4. History Historically the West Virginia Department of Transportation (WVDOT) has been utilizing geospatial (GIS, Remote Sensing, & GPS) technology for planning, analysis, and mapping purposes ever since the technology became available. In 2005 the WVDOT GIS Section was created to meet the growing needs for geospatial data and services in transportation. In 2007 the GIS Section was renamed as the GIS Unit and merged with the Highway Data Services (HDS) Unit, a program responsible for processing changes in roadway information, and the update and maintenance of the Roadway Inventory Log, to form the current Geospatial Transportation Information (GTI) Section.

  5. Program Planning & Administration Division Geospatial Transportation Information (GTI) Section Highway Data Services Unit GIS Unit - Geospatial Data Management -Geospatial Services • Roadway Inventory Log • -Roadway Statistics • -Public Certified Mileage

  6. GTI Mission & Vision Statements • Mission Statement • To improve customer service to the citizens of West Virginia by supplying the latest geospatial transportation information. • Vision Statement • The West Virginia Department of Transportation Geospatial Transportation Information (GTI) Section is committed to managing an enterprise Geospatial Information System that will improve the productivity of the West Virginia Department of Transportation, meet growing customer needs, and play a leading role in West Virginia’s geospatial community.

  7. GIS Implementation • Esri Shop • Enterprise License Agreement (ELA) • Esri Technology • Staff • 7 GIS Professionals • 2 Cartographers • 1 GIS Coordinator at District 7 • Enterprise GIS • WVDOT Enterprise GIS Roadmap • Enterprise Resource Planning (ERP)/Asset Management System

  8. Major GIS Projects in 2012 • Enterprise GIS Implementation • New Transportation GIS Data Model • District Level GIS Implementation • Workflow Manager • Transportation GIS Data Collection • (MS4, Railroad Crossings, Landslides, Outdoor Advertising, • Right-of-Way, Trails, Meta Data, etc.) • Highway Map Reproduction • Web Mapping Applications • (STIP, Park & Ride, Functional Classifications, etc.) • Mobile GIS Applications (MS4, etc.) • 3D GIS Application Development • GIS Portal Upgrade • ProjectWise Implementation • ERP/Asset Management System Implementation

  9. WVDOT Geospatial Data Repository

  10. Shapefiles for Download Source Data WVDOT Geospatial Data Repository • TRANSPORTATION • GIS DATA • Commissioner Orders • Paper Maps • Field Data • Consultant Reports • Etc. Editing Version - Edit GTI_EDITING Version - Edit Update Update Export Export Publication GTI_PUB_UTM Geospatial Analysis • REFERENCE • GIS DATA • WV GIS Tech Center • WVGES • USGS • Esri • Etc. Shapefiles Upload Upload Download &Add Metadata Download & Add Metadata Reproject Reproject Map Production Web Apps GTI_PUB_WM • IMAGERY • Aerial Photos • Shade Relief • Hillshade • Topo • Etc. Imagery GIS Services Store Store

  11. Automation Implementation

  12. Automation Implementation • Four Automated Data Management Operations • GTI_EDITING GDB  GTI_PUB_UTM GDB • Shapefiles GTI_PUB_UTM GDB • GTI_PUB_UTM GDB GTI_PUB_WM GDB • GTI_PUB_UTM GDB Shapefiles for Publications ArcGIS Model Builder GP1 Windows Scheduler Pythonwin GP2 Export Deploy Data Model Python Script Task GP … Improve GPn

  13. Key Issues & Solutions

  14. Database Lock • When a GDB is updated, it’s been accessed by clients (ArcGIS Desktop, ArcGIS Server, etc.). Database locks are created accordingly. • #Remove Database Locks on GTI_PUB_WM • Import pyodbc • cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=dotarcgis02;DATABASE=GTI_PUB_UTM;UID=xxxxx;PWD=xxx') • cursor = cnxn.cursor() • cursor.execute("delete FROM [GTI_PUB_UTM].[dbo].[SDE_process_information]" ) • cnxn.commit()

  15. Scheduled Task • After a Python script passes test, use Windows Scheduler to create a task so the script would run on a regular basis. • Make the task independent to individual’s login

  16. Email Notification? • If a task fails, notify the data repository manager by email • #Email • import smtplib, MIMEText • try: • … • except: • sender = 'admin@wvdohwebserver.com‘ • receivers = ['wuyueming@gmail.com', 'wu_yueming@yahoo.com'] • msg = MIMEText("The GTI_PUB_UTM_2_Web_Shapefiles script failed. Please check!") • msg['Subject'] = 'Python Script Failed' • msg['From'] = "Admin@WVDOHGISWebServer.com" • msg['Reply-to'] = "GTI Core Team" • msg['To'] = "Admin@WVDOHGISWebServer.com" • smtpObj = smtplib.SMTP('localhost') • smtpObj.sendmail(sender, receivers, msg.as_string())

  17. or Log • After a task is created, use a log file to track. • #Log • Import os, sys, time • rootFolder = r"C:\Geoprocessing Scripts\\" # This is where the log file is located • text_file = open(rootFolder + "PScriptLogFile.txt", "a") • text_file.write("\n\n") • text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py began at " + str(time.strftime('%X %x %Z')) + ".\n\n") • try: • … • text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py successfully ended at " + str(time.strftime('%X %x %Z')) + ".") • except: • text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py failed at " + str(time.strftime('%X %x %Z')) + ".") • text_file.close()

  18. Future Work

  19. Shapefiles for Download Source Data WVDOT Geospatial Data Repository • TRANSPORTATION • GIS DATA • Commissioner Orders • Paper Maps • Field Data • Consultant Reports • Etc. Editing Version - Edit GTI_EDITING Version - Edit Update Update Export Export Replicate Publication GTI_PUB_UTM Geospatial Analysis • REFERENCE • GIS DATA • WV GIS Tech Center • WVGES • USGS • Esri • Etc. Shapefiles Upload Upload Download &Add Metadata Download &Add Metadata Reproject Reproject Map Production Web Apps GTI_PUB_WM • IMAGERY • Aerial Photos • Shade Relief • Hillshade • Topo • Etc. Imagery GIS Services Store Store

  20. Headquarters Districts 1- 10 Editing GTI_EDITING Update Update Replicate Publication Publication GTI_PUB_UTM GTI_PUB_UTM FGDB Replicate Shapefiles Upload Upload Reproject Reproject GTI_PUB_WM Imagery WVDOT Geospatial Data Repository

  21. Yueming Wu GIS Manager WVDOT (304) 558-7437 yueming.wu@wv.gov

More Related