Updating - Methods and Best Practices
- DarkLight
Updating - Methods and Best Practices
- DarkLight
Article Summary
Share feedback
Thanks for sharing your feedback!
Overview
Matillion ETL regularly receives updates that include security enhancements, new features, new components, quality-of-life improvements, bug fixes, and more. Your client will usually alert you via the Notices tab when a new update is available.
We highly recommend updating immediately; however, this isn't mandatory. All software updates are inherently risky.
Matillion ETL integrates with over 100 external services, many of which are also updated regularly. Improvements and fixes included in the latest updates often risk causing issues with older versions of systems and/or services, therefore there is both a need to update and a risk in doing so.
Warning
Always back up your instance before performing any updates:
Before updating a Matillion ETL instance, please ensure that the hostname artifacts.matillion.com is whitelisted.
There are also CentOS and PostgreSQL repositories that are enabled on the instance by default. The CentOS repositories are mirrors that can change, so can be difficult to whitelist. You can disable or point to an internal repository instead if required.
Options for updating an instance
Matillion ETL supports two main options for updating the version of your instance:
- Migration, where data is copied from an existing instance (the source instance) and migrated to another instance (the target instance). The target instance is usually new.
- In-place update, where the updates are applied to the existing instance.
In most scenarios, the recommended approach is migration. This is considered the safest method, as the risk of running into issues is greatly reduced.
Pros and cons
Migrate
| Pros | Cons |
|---|---|
| The risk of causing disruption to a production instance is vastly reduced. By migrating to a new instance you can verify that your workloads run as expected on the new version of Matillion ETL, and only enable the scheduling on the new instance when you're happy (and then remove/retire the existing infrastructure). | Some elements, such as system modifications, aren't automatically migrated, as detailed in Migrate. Some of these items can be managed via the API, but not all. |
| You can test the new instances over a period of time while the old software is still running production jobs. | Requires additional effort from users to maintain a pre-production environment and perform testing (although this is good practice). |
| Matillion recommends that users manually maintain a record of any configuration changes made to your instances, ready for update, so that they can be manually recreated on a new instance, or where possible, those changes should be scripted so they're repeatable. | |
| It's not possible to use migrate in a high availability (HA) cluster. | |
| On rare occasions, components may be deprecated and replaced by one with a newer driver—known as "versioned" components in the release notes. In these cases, a migration update will render the old components (and thus any jobs they are a part of) inoperable as old drivers are not included in new instances. It is up to the user to replace the deprecated components with their updated counterparts. |
In-place update
| Pros | Cons |
|---|---|
| An update can be performed via the Matillion ETL user interface (UI) by a non-specialist. The RPM update is performed in the background and the service restarted. | While Matillion takes every precaution to ensure there are no problems, it's possible that an issue could arise which causes your instance to stop working. If this happens, there is no "undo" or revert option, which means you will need to restore from a backup. It's for this reason we strongly recommend that you should take backups (from within Matillion ETL or another method) of the virtual machine (VM) and ensure that you have tested that a working system can be restored from that backup, so you can recreate the instance if the update causes issues. |
| Doesn't require additional effort to set up a new instance and manage the connectivity between the current and new instance. | In-place updates can't always be used. For example, moving from Amazon Linux to CentOS is an operating system change that can never be offered as an in-place update and must instead be resolved via a migration. |
| Only the Matillion ETL application is updated. Postgres, Tomcat, and OS updates/upgrades aren't applied. |
Performing a migration (recommended method)
The main way to migrate your data from one instance to another is to use the Migrate feature in the project menu. However, it's also possible to create your own scripts using the API; while that can be quite an involved process, it can also be customized to your specific needs and made repeatable.
Performing an in-place update
This method can either be performed using the Matillion ETL UI, or via yum updates over SSH. For more information, read Updating - In-Place Updates.
Note
Please consider the following:
- Make sure you have taken a backup prior to making any changes.
- Make sure you are able to quickly restore from the backup in the event of any problems.
Updating HA clusters
Read Updating your Matillion HA Cluster for more information.
Updating server utilities and dependencies
We recommend that you apply critical security updates using the command yum update --security.
Applying only the latest security updates in this way shouldn't cause Matillion ETL features or functionality to break. However, the Matillion ETL application isn't tested with every minor update to all dependencies, so we don't suggest updating all packages via yum update. If you do so, it's at your own risk.
Prior to updating the Matillion ETL application or other yum packages, or applying any custom configuration, we always strongly suggest creating a root volume backup and being prepared to restore it to your virtual machine in the event of any loss of service or impact to Matillion ETL features and functionality.