API v1 - Git Integration
    • Dark
      Light

    API v1 - Git Integration

    • Dark
      Light

    Article Summary

    Overview

    This article is a guide to using the source control management (SCM) API in Matillion ETL. Matillion ETL uses Git for its version control, allowing teams to track changes (both small and large) to a project or a shared job with speed and efficiency.

    Note

    • Consider whether Git integration is appropriate for your Matillion ETL practices with our When to Choose Git article.
    • Git integration
    • Git FAQ
    • If you are new to Matillion ETL's API, read our Matillion API introduction.
    • Explore the v1 API map.
    • To jump to the v1 API, just run {host:port}/rest/v1 in your browser.
    • Git integration in Matillion ETL is an Enterprise Mode feature. Read about the benefits of upgrading your license to Enterprise Mode here.
    • All information in <angle brackets> should be replaced with information based on your own configuration.
    • The API has a different set of endpoints for working with (a) projects and (b) shared jobs. The examples in this article are for use with projects, with the endpoints needed for use with shared jobs listed in the table at the end of the article.


    Initialize A Local Repository

    To initialize a local repo using the Matillion v1 API, make the following POST call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/init

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Project/project/name/Docs/version/name/default/scm/init

    The variables in the above example are listed immediately below for added clarity. For security reasons, the instance address remains hidden.

    • groupName = Docs_Project
    • projectName = Docs
    • versionName = default

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    The table below documents the type of API endpoint, the body (in a JSON format), a description of the endpoint, and any pertinent notes that users should consider.

    MethodBody (JSON)Description
    POST
    {
    "commitMessage": "<Your Commit Message>",
    "userIdentifier": {
    "username": "<username>",
    "emailAddress": "<emailaddress>"
    }
    }
    
    Initializes a new repository in this Matillion ETL version.

    This request creates an initial Git commit within Matillion ETL.
    • commitMessage: A message to include with your commit. This will be visible on the commit inside the Matillion ETL client
    • username: The username to attach to the commit. Note this is not necessarily your username for the remote repository nor your Matillion ETL instance username and does not affect anything else.
    • emailAddress: The email address to attach to the commit (xxx@xxx.xxx). Note this is not necessarily your email for the remote repository nor your Matillion ETL instance email and does not affect anything else.

    Clone Remote Repository

    To clone a remote repository, make the following POST call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/clone

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/version/name/default/scm/clone

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    The table below documents the type of API endpoint, the body (in a JSON format), a description of the endpoint, and any pertinent notes that users should consider.

    MethodBody (JSON)Description
    POST
    {
    "remoteURI":"<URL>",
    "auth": {
    "authType": "HTTPS",
    "username": "<username>",
    "password": "<password>"
    }
    }
    
    or:
    {
    "remoteURI":"<URL>", 
    "auth": { 
    "authType": "SSH", 
    "privateKey": "<key>", 
    "passphrase": "<passphrase>"
    }
    }
    
    Clone an existing remote repository in this Matillion ETL version.

    Successful requests will initialise from or clone your remote repository.
    • remoteURI: The URL of your remote Git repository
    • authType: Matillion ETL currently supports HTTPS and SSH
    • username: The username to authenticate with the remote repository
    • password: The corresponding password for the above username
    • privateKey: An SSH private key
    • passphrase: The passphrase associated with the SSH private key

    Commit to Local Repository

    To commit changes to a local repo using the Matillion v1 API, make the following POST call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/commit

    Here is an example of this request with example variables added (for security, the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Project/project/name/Docs/version/name/default/scm/commit
    

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    SCM commit endpoint

    The table below documents the type of API endpoint, the body (in a JSON format), a description of the endpoint, and any pertinent notes that users should consider.

    MethodBody (JSON)Description
    POST
    {
    "branchName":"<branch>",
    "changes":[{"type":"ADDED","filePath":"<path>"}],
    "commitMetadata":{"commitMessage":"<Your Commit Message>","userIdentifier":{"username":"<username>","emailAddress":"<email address>"}}
    }

    Commit changes to a shared repository.

    • branchName: The repository branch that the changes will be committed to. E.g. "default".
    • filePath: A valid filepath to the changed objects being committed.
    • commitMessage: A message to include with your commit. This will be visible on the commit inside the Matillion ETL client.
    • username: The username to attach to the commit. Note this is not necessarily your username for the remote repository, nor your Matillion ETL instance username, and does not affect anything else.
    • emailAddress: The email address to attach to the commit (xxx@xxx.xxx). Note this is not necessarily your email for the remote repository, nor your Matillion ETL instance email, and does not affect anything else.



    Create Branch

    To create a new branch on the current commit, make the following POST call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/createBranch

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    {
    "httpMethod": "POST",
    "name": "Create new branch",
    "path": "/createBranch",
    "arguments": [
    {
    "style": "DATA",
    "name": "createBranchInfo",
    "type": "CreateBranchInfo"
    }
    ],
    "type": "String",
    "example": "/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/createBranch WITH POST DATA",
    "totalPath": "/group/name/{groupName}/project/name/{projectName}/version/name/{versionName}/scm/createBranch"
    }
    

    The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.

    MethodBody (JSON)Description
    POST
    {
    "branchName": "<name of new branch>"
    }
    
    Create a new branch on the current commit. The POST call will return the branch name along with the commit ID.
    • branchName: The name of the branch to be created.




    Switch Commit

    To switch to another Git commit using the API, make the following POST call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/switchCommit

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/version/name/default/scm/switchCommit

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.

    MethodBody (JSON)Description
    POST
    {
    "commitID": "<ID>"
    }
    
    Switch to an existing commit by passing the commit ID. You can find commit IDs by using the 'getState' endpoint detailed in the 'Get Commits' section of this article.
    • commitID: The commit ID of the commit to switch to

    Server Response

    A successful POST call to the endpoint will return the following response from the server:

    {
    "success": true,
    "result": "Success"
    }
    

    A failed call will return a brief error description, for example:

    {
    "success": false,
    "result": "Error while running command: An error occurred while switching commits\nCaused by: ab01418adceb9a6ec51e49ee7d7787dde74220e"
    }
    


    Fetch from Remote Repository

    To perform a fetch from a remote repository, make the following POST call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/fetch

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/fetch

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    The table below documents the type of API endpoint, the body (in a JSON format), a description of the endpoint, and any pertinent notes that users should consider.

    MethodBody (JSON)Description
    POST
    {
    "auth": {
    "authType": "HTTPS",
    "username": "<username>",
    "password": "<password>"
    },
    "fetchOptions": {
    "removeDeletedRefs": "<true/false>",
    "thinFetch": "<true/false>"
    }
    }
    
    or
    {
    "auth": { 
    "authType": "SSH", 
    "privateKey": "<key>", 
    "passphrase": "<passphrase>"
    },
    "fetchOptions": {
    "removeDeletedRefs": "<true/false>",
    "thinFetch": "<true/false>"
    }
    }
    Fetch from a remote repository.

    A remote repository must be configured before fetch requests can be made.
    • authType: Matillion ETL currently supports HTTPS and SSH
    • username: The username to authenticate with the remote repository
    • password: The corresponding password for the above username
    • privateKey: An SSH private key
    • passphrase: The passphrase associated with the SSH private key
    • removeDeletedRefs: Choose to remove deleted references ("true" or "false")
    • thinFetch: Reduces the data sent when the sender and receiver share many of the objects ("true" or "false")


    Push to a Remote Repository

    To push to a remote repository, make the following POST call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/push

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/push

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.

    MethodBody (JSON)Description
    POST
    {
    "auth": {
    "authType": "HTTPS",
    "username": "<username>",
    "password": "<password>"
    },
    "pushOptions": {
    "atomic": "<true/false>",
    "forcePush": "<true/false>",
    "thinPush": "<true/false>"
    }
    }
    
    or
    {
    "auth": {
    "authType": "SSH", 
    "privateKey": "<key>", 
    "passphrase": "<passphrase>"
    },
    "pushOptions": {
    "atomic": "<true/false>",
    "forcePush": "<true/false>",
    "thinPush": "<true/false>"
    }
    }
    
    Push to a remote repository.
    • authType: Matillion ETL currently supports HTTPS and SSH
    • username: The username to authenticate with the remote repository
    • password: The corresponding password for the above username
    • privateKey: An SSH private key
    • passphrase: The passphrase associated with the SSH private key
    • atomic: Guarantees that either all references will be pushed on the remote, or none of them will; this option avoids partial pushes. ("true" or "false"),
    • forcePush": Forces the local revision to be pushed into the remote repository. This action can cause the remote repository to lose commits, and should be used with caution.("true" or "false")
    • thinPush": Reduces the data sent when the sender and receiver share many of the objects. ("true" or "false")


    Configure a Remote Repository

    To configure a remote repository, make the following POST call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/remote

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/remote

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.

    MethodBody (JSON)Description
    POST
    {
    "url": "{your_remote_uri}"
    }
    
    Configure a remote repository.
    • url: The URL of your remote Git repository.


    Get Remote (returns URL)

    To return the URL of your remote repository, make the following GET call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/remote

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/remote

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:



    Get Commits

    To retrieve a list of each Git commit on the local repository, make the following GET call:

    <instance address>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/getState

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/getState

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.

    MethodBody (JSON)Description
    GETN/AProvides a list of commits on the local repository.


    Get Differences

    To get differences between the current Matillion version and the branch head, make the following GET call:

    <instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/differences

    Here is an example of this request with variables added (the instance address has not been added):

    {host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/version/name/default/scm/differences

    Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:

    The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.

    MethodBody (JSON)Description
    GETN/AReturns the difference between the current version (including uncommitted changes) and the head.

    The following is an example of the server response to the GET call:

    {
    "success": true,
    "result": {
    "branches": [
    "master"
    ],
    "changes": {
    "dotfileChanges": [],
    "nonDotfileChanges": [
    {
    "type": "MODIFIED",
    "filePath": "ROOT/o1.ORCHESTRATION"
    },
    {
    "type": "MODIFIED",
    "filePath": "version"
    }
    ]
    }
    }
    }

    The type response can be ADDED, REMOVED, or MODIFIED.



    Shared Jobs

    Many of the functions listed above can also be used with shared jobs, by making a POST call with the form:

    <instanceAddress>/rest/v1/git/shared-jobs/<endpoint>

    The endpoints which can be used for shared jobs are listed in the following tables.

    FunctionMethodEndpointPOST Body
    Initialize shared jobs repositoryPOST
    <instanceAddress>/rest/v1/git/shared-jobs/init
    See Initialize A Local Repository
    Clone remote repository for shared jobsPOST
    <instanceAddress>/rest/v1/git/shared-jobs/clone
    See Clone Remote Repository
    Commit shared job changes to local repositoryPOST
    <instanceAddress>/rest/v1/git/shared-jobs/commit
    See Commit to Local Repository
    Switch to another shared job commitPOST
    <instanceAddress>/rest/v1/git/shared-jobs/switchCommit
    See Switch Commit
    Fetch shared job from remote repositoryPOST
    <instanceAddress>/rest/v1/git/shared-jobs/fetch
    See Fetch from Local Repository
    Push shared job to remote repositoryPOST
    <instanceAddress>/rest/v1/git/shared-jobs/push
    See Push to a Remote Repository
    Get differences between the current Shared Job version and the branch headGET
    <instanceAddress>/rest/v1/git/shared-jobs/differences
    N/A
    Set remote url for git shared jobs repositoryPOST
    <instanceAddress>/rest/v1/git/shared-jobs/remote
    {
    "url": "<URL>"
    }
    
    Create a new branch on current commitPOST
    <instanceAddress>/rest/v1/git/shared-jobs/branch
    {
    "branchName": "<name of new branch>"
    }
    
    Return the URL of the remote repositoryGET
    <instanceaddress>/rest/v1/git/shared-jobs/remote
    N/A
    Return the history of commits in all branches of the repositoryGET
    <instanceaddress>/rest/v1/git/shared-jobs/state/<max_count>/<skip>

    • max_count: the number of commits to be returned.
    • skip: the number of commits to skip before returning the <max_count> commits.
    N/A
    Delete a branchDELETE
    <instanceaddress>/rest/v1/git/shared-jobs/branch/<branchName>
    • branchName: the name of the repository branch to be deleted.
    N/A
    Delete the repositoryDELETE
    <instanceaddress>/rest/v1/git/shared-jobs
    N/A
    Merge a branch, assuming there are no conflicts. If conflict occurs, the server returns a list of conflicts. If the request contains conflict resolutions in the conflictsResolution field of the POST Body, then the resolutions are applied.POST
    <instanceaddress>/rest/v1/git/shared-jobs/merge
    See below

    The /merge endpoint has the following POST Body:

    {
    "commitMetadata": {
    "commitMessage": "<message>",
    "userIdentifier": {
    "username": "<name>",
    "emailAddress": "<email>"
    }
    },
    "checkoutAfterMerge": "<true | false>",
    "mergeFrom": "<source branch>",
    "mergeInTo": "<target branch>",
    "conflictsResolution": [
    {
    "package":"<package>",
    "name": "<name>",
    "revision": <number>,
    "choice" : "<OURS | THEIRS>"
    }
    ]
    }


    Contact Us

    If any help is needed navigating the Git API in Matillion ETL, please read Getting Support.