Migrate
    • Dark
      Light

    Migrate

    • Dark
      Light

    Article Summary

    Overview

    The Migrate feature in Matillion ETL enables you to copy resources from one ETL instance to another. Migrate is a distinct and separate process to Updating. There may be occasions where it's preferable to migrate to a new instance rather than update your current one, such as when a new version of Matillion ETL is deployed. Read Updating - Methods and Best Practices for specific information around these two processes.

    Note

    Please read Preparing to Migrate if you've not already done so.

    Please make yourself aware of the following principles:

    • You need at least two ETL instances for a migration. Your original instance—which contains the resources you want to copy (referred to as the Source instance)—and a new instance to copy those resources into (referred to as the Target instance).
    • A migration is not a 100% duplication of one instance into another. Only specified resources are copied to the target instance and you'll need to recreate any non-migratable resources. The following resources cannot be migrated:
      • Custom licences.
      • SSL certificates.
      • Task history.
      • Python libraries.
      • Non-standard JDBC drivers (click Admin in the top-right of the user interface (UI), and then click Manage Database Drivers to view the list of database drivers that come with Matillion ETL "out of the box").
      • Customized "standard" API profiles.
      • LDAP setup.
      • Backend configuration changes (for example: emerald.properties file changes).
    • A new ETL instance is not created as part of this process. You must have the new instance ready and operational before you begin.
    • No resources in the source instance are deleted or otherwise altered as a result of the migration.
    • Users should not migrate from a DBT-enabled instance to a non-DBT-enabled instance.
    Note
    • It's recommended that you configure any non-migratable resources on your target instance before the migration.
    • If you rely on operating system (OS) customizations, such as additional libraries or drivers, please consider creating your own AMI based on ours with your customizations "baked in". Then, upon each release, remake your AMI from our latest one and test. This could also be used to ensure that drivers, user configuration, and so on, are "baked in".

    Migrating Git resources

    As of version 1.61, Matillion ETL supports the migration of Git resources. You can choose to migrate project content with or without Git, as described below.
    When you perform a migration with Git enabled, commits for orchestration and transformation jobs in every project in the target instance will be flagged as Modified. You will need to perform a one-time commit of all jobs after the migrate, which will remove the Modified flag.

    This is expected behavior, due to some of the formats of the jobs changing during the upgrade, and is not a cause for concern. These are valid changes to the structure of the jobs and you can commit them safely.

    To learn more about Git integration within Matillion ETL, read Git Integration with Matillion ETL.


    Connection

    In Matillion ETL, click AdminMigrate.

    Configure the details of the target instance.

    PropertyDescription
    Target instance URLThe URL of the Matillion ETL instance you wish to migrate resources to. You must include the "http://" or "https://" portion of the URL.
    UsernameYour Matillion ETL username for the target instance.
    PasswordYour Matillion ETL password for the target instance.

    Click Test Connection to confirm that your details are correct. You are not able to continue if the test fails. Once the test is completed successfully, click Next to continue.


    Users

    Tick the Migrate all users, groups and permissions checkbox if you wish to migrate user configuration settings, including users, user groups, usernames, passwords, roles, and permissions.

    Warning

    All user, group, and permissions data will be overwritten in the target instance if it already exists.

    Users who wish to migrate user resources must have the role Server Admin for their source user and their target user.

    A user's username and password are securely migrated from source to target.

    LDAP must be set up in the target instance before the migration. Read LDAP Integration to learn more.

    Click Next to move to the next page.


    Project Content

    Use the file explorer to select the project content you wish to include in the migration.

    Note
    • If migrating only part of a project group or project, make sure that the relevant hierarchy exists on the target instance. For example, if migrating a version, the project and project group must already exist.
    • If you're migrating project content, the source and target users must have the Global Project Admin role in Matillion ETL. See User Configuration for details of how to do this.
    • Projects are filtered based on an access control list, and you will not be able to migrate a project unless you can access it on both the source and target instances.

    If your Matillion ETL instance has projects that use Git, this page of the wizard will ask How would you like to migrate projects which use Git? Select either Migrate with Git or Migrate without Git. The projects that are using Git are identified in the wizard with a Git icon next to the name.

    Migrating projects with Git will migrate the local Git repository, including remote repository configuration if it exists.

    Warning

    Any uncommitted changes are not migrated.

    Click Next to move to the next page.


    Instance configuration

    Use the file explorer to select the instance content you wish to include in the migration.

    Click Next to move to the next page.

    Migration Options

    Choose additional migration options.

    • Disable schedules: Disables the scheduler on the target instance. This is selected by default. This will not affect individual schedules, which will retain their Enabled/Disabled status, but schedules won't run until the scheduler is re-enabled. Users can re-enable the scheduler at their own discretion.
    • Disable queues: Disables queues on the target instance. This is selected by default. Users can re-enable their queues at their own discretion.
    • Disable CDC: Disables Change Data Capture (where available) on the target instance. This is selected by default. Users can re-enable CDC at their own discretion.
    • Disable CDC task source creation: Disables CDC task source creation (where available) on the target instance. This is selected by default.
    • Delete project if already exists: Select this if you are migrating a project that already exists on the target instance in some form and you wish to delete the existing version of the project on the target instance. This isn't selected by default.

    When you are happy with the configuration, click Migrate to begin the migration process.

    On completion, a Migrate dialog shows the status of the migration, listing each migrated element with a "Success" message or error message if it has failed.