1 / 39

Using the Migrate Module

Using the Migrate Module. How to get legacy data into Drupal. Introduction and Overview. Dwayne Bailey Eye Street Solutions dwayne.bailey@eyestreet.com 609-815-0787 dbcollies Migrate module concepts Example Migration Resources Questions. Migrate module concepts. Very flexible

dextra
Download Presentation

Using the Migrate Module

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using the Migrate Module How to get legacy data into Drupal

  2. Introduction and Overview • Dwayne BaileyEye Street Solutionsdwayne.bailey@eyestreet.com609-815-0787dbcollies • Migrate module concepts • Example Migration • Resources • Questions

  3. Migrate module concepts • Very flexible • But at the cost of custom module development • Allows for migrations to be stopped and restarted • Migrations can be rolled back • Manage deployments from a UI or via drush • Create a module which subclasses DynamicMigration • Add a hook to tell Migrate about the class • Define the source of the migration • Define the destination • Map fields in the source to fields in the destination • How to keep track of what has been migrated • Migrate assumes one row of the source maps to one row of the destination

  4. Migrate hook • Defined in file modulename.migrate.inc function modulename_migrate_api() { $api = array( 'api' => 2, 'migrations' => array( ’Migration1' => array('class_name' =>’MigrationClass1'), ’Migration2' => array('class_name' =>’MigrationClass2’) ’Migration3' => array('class_name' =>’MigrationClass3’) ); return $api; }

  5. DynamicMigration class abstract class BasicAppealMigration extends DynamicMigration { public function __construct() { // Always call the parent constructor first for basic setup parent::__construct(); } } class Migration1 extends BasicAppealMigration { public function __construct() { parent::__construct(); $this->description = t(’A human readable description'); $this->dependencies = array(‘Migration2’, ‘Migration3’);

  6. DynamicMigration class $this->source = new MigrateSourceSQL(…); $this->destination = new MigrateDestinationNode(…); $this->map = new MigrateSQLMap(…); $this->addFieldMapping(…);

  7. Migration Sources • MigrateSourceCSV • MigrateSourceJSON • MigrateSourceMSSQL • MigrateSourceOracle • MigrateSourceSQL • MigrateSourceXML • ???

  8. Migration Destinations • MigrateDestinationComment • MigrateDestinationMenu • MigrateDestinationMenuLinks • MigrateDestinationNode • MigrateDestinationTable • MigrateDestinationTerm • MigrateDestinationUser • ???

  9. Migration Mapping • MigrateSQLMap new MigrateSQLMap($this->machineName, array( 'CATEGORY_ID' => array('type' => 'int', 'not null' => TRUE, 'unsigned' => TRUE, 'description' => 'Category ID', ) ), MigrateDestinationTerm::getKeySchema() );

  10. Field Mapping $this->addFieldMapping(‘destfield’, ‘sourcefield’) ->description(t(‘A description of this mapping’)) ->issueGroup(t(‘A Grouping Label’)) ->issuePriority (MigrateFieldMapping::ISSUE_PRIORITY_MEDIUM) ->issueNumber(770064) ->separator(‘,’) ->sourceMigration(‘someOtherMigration’) ->arguments(array(‘source_type’ => ‘tid’)) ->defaultValue(‘Some Default here’); $this->addFieldMapping(‘destfield2’) ->issueGroup(t(‘No source data available’)); $this->addFieldMapping(NULL, ‘sourcefield2’) ->issueGroup(t(‘Not migrated from source));

  11. Migrate UI Usage

  12. Migrate Drush commands migrate-import (mi) Perform one or more migration migrate-reset-status(mrs) Reset a active migration's status migrate-rollback (mr) Roll back the destination objects migrate-status (ms) List all migrations with current status. migrate-stop (mst) Stop an active migration operation migrate-wipe (mw) Delete all nodes from specified content types.

  13. Example Migration • Source: Oracle database • Usable Source: CSV Files • Destination: • Custom Content type • New taxonomy vocabulary • Intermediate database tables

  14. Source Schema

  15. Target Content Type

  16. Generic Parent Migration abstract class BasicAppealsMigration extends DynamicMigration { public function __construct() { // Always call the parent constructor first for basic setup parent::__construct(); } }

  17. Taxonomy Term Migration class AppealTermMigration extends BasicAppealsMigration { public function __construct() { parent::__construct(); $this->description = t('Migrate Appeal Categories from the source CSV to taxonomy terms');

  18. Taxonomy Term Migration $this->map = new MigrateSQLMap($this->machineName, array( 'CATEGORY_ID' => array('type' => 'int', 'not null' => TRUE, 'unsigned' => TRUE, 'description' => 'Category ID', ) ), MigrateDestinationTerm::getKeySchema() );

  19. Taxonomy Term Migration $this->source = new MigrateSourceCSV( variable_get('migrate_appeals_categories_csv'), array( array('CATEGORY_ID', 'Category ID'), array('CATEGORY_DESCRIPTION', 'Category Name'), ), array( 'header_rows'=>variable_get ('migrate_appeals_categories_header') ) );

  20. Taxonomy Term Migration $this->destination = new MigrateDestinationTerm('appeals_category'); $this->addFieldMapping('name', 'CATEGORY_DESCRIPTION'); $this->addFieldMapping('description') ->issueGroup(t('DNM')); $this->addFieldMapping('format') ->issueGroup(t('DNM')); $this->addFieldMapping('weight') ->issueGroup(t('DNM')); $this->addFieldMapping('parent') ->issueGroup(t('DNM'));

  21. Appeal Node Migration? • Multiple source rows map to a single destination node • To simplify processing, intermediate database tables are created prior to the final migration

  22. AppealDocument Migration class AppealDocTableMigration extends BasicAppealsMigration { public function __construct() { parent::__construct(); $this->description = 'Miscellaneous table data'; $table_name = 'migrate_appeals_documents';

  23. AppealDocument Migration $this->map = new MigrateSQLMap($this->machineName, array('APPEAL_ID' => array( 'type' => 'int', 'unsigned' => TRUE, 'not null' => TRUE, ), 'TYPE' => array( 'type' => 'varchar', 'length' => 1, 'not null' => TRUE, ) ), MigrateDestinationTable::getKeySchema($table_name) );

  24. AppealDocument Migration $this->source = new MigrateSourceCSV( variable_get('migrate_appeals_documents_csv'), array( array('APPEAL_ID', 'Appeal ID'), array('TYPE', 'Appeal Type'), array('DOCUMENT', 'Document'), ), array( 'header_rows' => variable_get ('migrate_appeals_documents_header'), 'embedded_newlines' => TRUE ));

  25. AppealDocument Migration $this->destination = new MigrateDestinationTable($table_name); $this->addFieldMapping('appeal_id', 'APPEAL_ID'); $this->addFieldMapping('appeal_type', 'TYPE'); $this->addFieldMapping('document', 'DOCUMENT');

  26. Other Table Migrations • AppealCategoryTable • AppealNodeTable

  27. AppealNode Migration class AppealNodeMigration extends BasicAppealsMigration { public function __construct() { parent::__construct(); $this->description = t('Appeals'); $this->dependencies = array('AppealTerm', 'AppealNodeTable', 'AppealCategoryTable', 'AppealDocTable');

  28. AppealNode Migration $this->map = new MigrateSQLMap($this->machineName, array( 'appeal_id' => array( 'type' => 'int', 'not null' => TRUE, 'description' => 'Appeal ID.', 'alias' => 'a', ) ), MigrateDestinationNode::getKeySchema() );

  29. AppealNode query $query = db_select('migrate_appeals_node', 'a') ->fields('a’, array('appeal_id', 'title', 'appeal_type', 'applicant_name', 'disaster_number', 'pa_id', 'date_signed', 'dsr', 'report_type')); $query->condition('deleted', 0);

  30. AppealNode Migration $query->leftJoin('migrate_appeals_documents', 'da', "a.appeal_id=da.appeal_id AND da.appeal_type='A'"); $query->leftJoin('migrate_appeals_documents', 'dl', "a.appeal_id=dl.appeal_id AND dl.appeal_type='L'"); $query->leftJoin('migrate_appeals_documents', 'ds', "a.appeal_id=ds.appeal_id AND ds.appeal_type='S'");

  31. AppealNode Migration $query->leftJoin('migrate_appeals_category_map', 'cm', 'a.appeal_id=cm.appeal_id'); $query->groupBy('a.appeal_id'); $query->addExpression('GROUP_CONCAT(CAST(cm.category_id as CHAR))', 'term_ids'); $query->fields('da', array('document')); $query->fields('dl', array('document')); $query->fields('ds', array('document'));

  32. AppealNode Migration $count_query = db_select('migrate_appeals_node', 'a'); $count_query->addExpression('COUNT(appeal_id)', 'cnt'); $count_query->condition('deleted', 0);

  33. AppealNode Migration $this->source = new MigrateSourceSQL( $query, // The base query array(), // Descriptions of fields $count_query, // A query to count the number of rows array('cache_counts' => TRUE) // Arguments );

  34. AppealNode Migration $this->destination = new MigrateDestinationNode('appeal'); $this->addFieldMapping('title', 'title') ->description(t('Mapping project title to node title')); $this->addFieldMapping('field_appeal_type', 'appeal_type'); $this->addFieldMapping('field_applicant_name', 'applicant_name'); $this->addFieldMapping('field_disaster_number', 'disaster_number'); $this->addFieldMapping('pa_id', 'pa_id'); $this->addFieldMapping('field_date_signed', 'date_signed'); $this->addFieldMapping('field_dsr', 'dsr'); $this->addFieldMapping('field_report_type', 'report_type');

  35. AppealNode Migration $this->addFieldMapping('field_analysis', 'document'); $this->addFieldMapping('field_analysis:format')->defaultValue('filtered_html'); $this->addFieldMapping('field_letter', 'dl_document'); $this->addFieldMapping('field_letter:format')->defaultValue('filtered_html'); $this->addFieldMapping('field_summary_brief', 'ds_document'); $this->addFieldMapping('field_summary_brief:format')->defaultValue('filtered_html');

  36. AppealNode Migration $this->addFieldMapping('appeal_categories', 'term_ids') ->separator(',') ->arguments(array('source_type' => 'tid')) ->sourceMigration('AppealTerm'); $this->addFieldMapping('appeal_categories:create_term') ->defaultValue(FALSE); $this->addFieldMapping('appeal_categories:ignore_case') ->defaultValue(FALSE);

  37. AppealNode Migration $this->addFieldMapping('sticky') ->defaultValue(1); $this->addFieldMapping('uid') ->defaultValue(1); $this->addUnmigratedDestinations( array('created', 'changed', 'status', 'promote', 'revision', 'language', 'revision_uid', 'log’, 'tnid’ ) );

  38. Resources • Migrate module: http://drupal.org/project/migrate • Migrate documentation: http://drupal.org/node/415260 • Example code: http://dwayne.thebaileys.name/appeals/migrate_appeals.tar • Slideshow: http://dwayne.thebaileys.name/appeals/migrate.ppsx • Contact info: Dwayne Bailey Eyestreet Solutions Dwayne.bailey@eyestreet.com 609-815-0787

  39. Questions?

More Related