1 / 63

Performing Data Migration

Performing Data Migration. USACE SDSFIE Training Prerequisites: Implementing a Local Adaptation. Video sequence. Objectives. Understanding the data migration process Understanding how to prepare for data migration Understanding the ESRI-based options for the data migration process

evita
Download Presentation

Performing Data Migration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Performing Data Migration USACE SDSFIE Training Prerequisites: Implementing a Local Adaptation

  2. Video sequence

  3. Objectives • Understanding the data migration process • Understanding how to prepare for data migration • Understanding the ESRI-based options for the data migration process • Understanding how to use the Simple Data Loader for semi-automated data migration • Understanding how to use the PLTS Data Loader option to perform automated data migration • Understanding problems with data loading process, and how to address them

  4. An overview of the data migration process

  5. The SDSFIEData Migration Process Source Data Dictionary Current District Source Data Set Crosswalk Tables Data Migraton Target Data Dictionary Future District Target Data Set

  6. The SDSFIEData Migration Process The Crosswalk plays a key role in data migration by guiding the data loading. There are typically up to 3 levels of Crosswalk tables: • Feature Class • Attribute • Enumerant Our demonstrations use 2 levels for simplicity Current District Source Data Set Crosswalk Tables Data Migraton Future District Target Data Set

  7. Automated Data Loading • Process moves through the highest level table, feature class by feature class • Each feature class mapping pair is referenced, one-by-one Source Data Set Target Data Set

  8. Automated Data Loading When migrating a specific feature class mapping pair (ID=2), the software accesses the set of attribute mapping pairs (IDs = 9 – 16) unique to that feature class mapping pair Source Data Set Target Data Set

  9. Automated Data Loading For attribute mapping pairs with a source and target enumeration, the software accesses the set of enumerant mapping pairs (IDs = 1 – 3) unique to that attribute pair (ID = 11) Source Data Set Target Data Set

  10. Preparing for data migration

  11. Migration Preparation Checklist • Source geodatabase (populated), consistent with the Source Data Dictionary used in the Crosswalk creation process • Target geodatabase (empty), compliant to your adaptation schema, based on the Target side of your Crosswalk • Crosswalk, with element-level mappings between your Source and your Target schemas • Software to support the data loading process • Knowledge base and/or support for any issues you may encounter

  12. ESRI based options for data migration

  13. ESRI Based Data Loading • Production Line Tool Set (PLTS) • Fully automated data loading, run from ArcCatalog • Requires simple reformatting of your Crosswalk tables • Best for data migrations involving many feature classes • Handles enumerant mappings, for attributes with enumerations • Requires additional license for extension • Simple Data Loader • Semi-Automated data loading, run from ArcCatalog • Crosswalk tables used as reference for attribute mappings • Best for data migrations involving relatively few feature classes • Does not require additional license, but does not handle domain mappings • Others • ArcMap Object Loader, run from in edit session • Safe Software FME (Feature Manipulation Engine)

  14. Using the simple data loader

  15. Using the Simple Data Loader • Right click target feature class • Select Load > Load Data • Press the Nextbutton • Click folder icon: • Navigate to and select source feature class, then press Open • Press the Add button • Check for file accuracy • Press the Next button • In attribute mapping window: • Check each Source field, and if incorrectly matched or unmatched: • Click in that Source field • Select correct match from list • Press the Next button, when ready • Press the Nextbutton • Read Summary, and press Finish No data present in Preview of target feature class!

  16. Using the Simple Data Loader • Select Load > Load Data

  17. Using the Simple Data Loader • Press the Nextbutton

  18. Using the Simple Data Loader • Click folder icon:

  19. Using the Simple Data Loader • Navigate to and select source feature class, then press Open

  20. Using the Simple Data Loader • Press the Add button

  21. Using the Simple Data Loader • Check for file accuracy, and then press the Next button

  22. Using the Simple Data Loader • Press the Next button

  23. Using the Simple Data Loader • In attribute mapping window: • Check each Source field, and if incorrectly matched or unmatched: • Click in that Source field

  24. Using the Simple Data Loader Use your attribute Crosswalk Tables to make the correct selection! • Check each Source field, and if incorrectly matched or unmatched: • Select correct match from list

  25. Using the Simple Data Loader • Press the Next button, when ready

  26. Using the Simple Data Loader • Press the Nextbutton

  27. Using the Simple Data Loader • Read Summary, and press Finish

  28. Using the Simple Data Loader • Repeat steps, starting with next target feature class Data now present in Preview of target feature class!

  29. Using the PLTS Data loader tool

  30. The PLTS Crosswalk Tables • PLTS requires a special format for Crosswalk tables • Excel workbook format, “XREF.xlsx” • Each XREF workbook contains 3 spreadsheets: • Dataset Mapping – source to target feature classes • Field Mapping – source to target attributes • Update Values – source to target enumerants (domain values) • Reformatting is very straightforward

  31. PLTS Preparation: Reformatting Crosswalk Tables • Download PLTS_XREF_Template.xlsx accessed through USACE component page of sdsfieonline.org • Download PLTS_XREF_Example.xlsx (same location) • Open _Example workbook, and familiarize yourself with structure • Open _Template workbook • Open your Crosswalk file • Before starting, note that template headings (Row 1) in each of the 3 spreadsheets cannot be altered!

  32. Reformatting Crosswalk Tables: Copying and Pasting • Start with Feature Type mappings in your Crosswalk file • Note, again, that template headings (Row 1) cannot be altered! • Select and Copy block of feature types from Source Name column • Click in Cell B2 in XREF Dataset Mappings sheet, and Paste • Select and Copy block of feature types from Target Name column • Click in Cell C2 in XREF Dataset Mappings sheet, and Paste • Repeat process for Attribute mappings spreadsheet, pasting into Field Mappings spreadsheet (into Cells C2 and D2, for source and target attributes, respectively) • Repeat process for enumerants, pasting into Update Values spreadsheet (into Cells C2 and D2, for source and target enumerants, respectively)

  33. Reformatting Crosswalk Tables: Copying and Pasting District Crosswalk workbook Do not copy headers! PLTS XREF workbook

  34. Reformatting Crosswalk Tables: Understanding Table Hierarchy • Look at _Example workbook • Pick a feature class mapping, noting the ID number • Find the set of SourceField attributes in the Field Mapping sheet that have the same ID number (in the DatasetMappingId field) as the feature class you picked; note this relationship • Pick an attribute mapping that involves a enumeration (domain), and note the ID number • Find the set of FromValue records in the Update Values sheet that have the same ID number (in the FieldMappingId field) as the attribute you picked; note this relationship

  35. Reformatting Crosswalk Tables: Updating the IDs • Dataset Mapping spreadsheet • Make sure every feature class mapping row has an ID, and that these numbers start in Cell A2 with “1” and continue sequentially downward • Field Mapping spreadsheet • Make sure every attribute mapping row has an ID, and that these numbers start in Cell A2 with “1” and continue sequentially downward • Fill in DatasetMappingId values for correct feature type (hint: paste in the feature type names from your Crosswalk, and then convert to the correct numbers)

  36. Reformatting Crosswalk Tables: Updating the IDs • Update Values spreadsheet • Make sure every eunumerant mapping row has an ID, and that these numbers start in Cell A2 with “1” and continue sequentially downward • Fill in FieldMappingId values for correct attribute (hint: paste in the attribute names from your Crosswalk, and then convert to the correct numbers) • Again, examine structure in the _Example file, and verify that you have done this correctly • Also, revisit slides 7 – 9 in this Video!

  37. The Three PLTS Tools • PLTS Create Cross-reference • Creates cross-reference in a personal geodatabase (GDB) • Utilizes the reformatted Crosswalk, now XREF.xlsx • Needs to know names and locations of source and target GDB • PLTS Preload Validate • Checks validity of all inputs to the data loading process • Needs to know names and locations of source and target GDB • Needs to know name and location of cross-reference PGDB • Outputs text file with summary information • PLTS Data Loader • Loads the data from source GDB to target GDB • Needs to know names and locations all 3 inputs • Outputs text file with loading success record counts (of total records) for each feature class

  38. Using the PLTS Data Loader Conditions for Starting PLTS sequence • Populated Source data present • Empty Target data present • XREF.xlsx present

  39. Using the PLTS Data Loader PLTS Create Cross-reference Tool • Press PLTS Create Cross-reference tool button

  40. Using the PLTS Data Loader PLTS Create Cross-reference Tool • Press “… “ button to navigate to and select source GDB

  41. Using the PLTS Data Loader PLTS Create Cross-reference Tool • Select source GDB, & press Select

  42. Using the PLTS Data Loader PLTS Create Cross-reference Tool • Press “… “ button to navigate to folder in which Cross-reference PGDB will be created

  43. Using the PLTS Data Loader PLTS Create Cross-reference Tool • Name PGDB, and press Save

  44. Using the PLTS Data Loader PLTS Create Cross-reference Tool • Specify Target and Mapping File • Repeat specification for Target GDB • Check Import mapping from file box • Repeat specification for XREF.xlsx • Check all four file specifications, and then press OK

  45. Using the PLTS Data Loader PLTS Create Cross-reference Tool • Press OK

  46. Using the PLTS Data Loader PLTS Create Cross-reference Tool Cross-reference PGDB has been created

  47. Using the PLTS Data Loader PLTS Preload Validate Tool • Press PLTS Preload Validate tool button

  48. Using the PLTS Data Loader PLTS Preload Validate Tool • Press “… “ button to navigate to folder in which Log File will be created

  49. Using the PLTS Data Loader PLTS Preload Validate Tool • Specify name for file, & press Save

  50. Using the PLTS Data Loader PLTS Preload Validate Tool • Specify other file inputs • Repeat specification for Target GDB • Repeat specification for Source GDB • Repeat specification for Cross-reference PGDB • Check all four file specifications, and then press OK

More Related