MindSphere Extract

Dark
Light

Article Summary

Share feedback

Thanks for sharing your feedback!

MindSphere Extract

The MindSphere Extract component calls the MindSphere API to retrieve and store data to be either referenced by an external table or loaded into a table, depending on the user's cloud data warehouse. Users can then transform their data with the Matillion ETL library of transformation components.

Using this component may return structured data that requires flattening. For help with flattening such data, we recommend using the Nested Data Load Component for Amazon Redshift and the Extract Nested Data Component for Snowflake or Google BigQuery.

Note: The tokens loaded from the MindSphere API have a lifespan of 30 minutes. These tokens are stored within an in-memory cache that expires each entry after 25 minutes have elapsed. At this point, a new token is requested.

Properties

Property	Setting	Description
Name	String	A human-readable name for the component. If a job includes more than one MindSphere Extract component, users might wish to name each component based on its data source.
API	Select	Select which MindSphere API to make queries to.
Data Source	Select	Select the data source. The available data sources depend on the chosen API. Upon selection, any properties specific to the chosen data source will be available for users to configure. This property is only available when the API property is set to "Asset Management EU1" or "Event Management EU1".
API Username	String	Input the API username for the chosen MindSphere app. For help locating your API Username, please refer to our MindSphere Extract Authentication Guide.
API Password	String	Input the API password for the chosen MindSphere app. For help locating your API password, please refer to our MindSphere Extract Authentication Guide. We recommend storing passwords in the Matillion ETL Password Manager.
App Name	String	Specify the name of target MindSphere application. Users can see an overview of their MindSphere apps, and create new apps, in the Developer Cockpit interface.
App Version	Version Number	Specify the version number of the target MindSphere application. Each application in the MindSphere Developer Cockpit interface has a version.
Host Tenant	String	Specify the Host Tenant. Users can find this credential in the MindSphere Asset Manager interface.
User Tenant	String	Specify the User Tenant. For help creating a user account or assigning MindSphere permissions, visit this page.
Entity Id	String	Specify your Entity Id. The Entity Id must exist in the IoT Entity Service. This property is only available when the Data Source is set to "IoT Time Series EU1".
Property Set Name	String	Specify the name of the Property Set. This property is only available when the Data Source is set to "IoT Time Series EU1".
Select	String	Specify the data to select from the Property Set. This property is only available when the Data Source is set to "IoT Time Series EU1".
Latest Value	Boolean	When True, Matillion ETL will only return the most recent value recorded for the metrics requested. This property is only available when the Data Source is set to "IoT Time Series EU1".
From	Timestamp	Specify the start point of the desired time range in the API call. The format is `YYYY-MM-DDThh:mm:ss.mmmZ` where `T` is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the Data Source is set to "IoT Time Series EU1".
To	Timestamp	Specify the end point of the desired time range in the API call. The format is `YYYY-MM-DDThh:mm:ss.mmmZ` where `T` is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the Data Source is set to "IoT Time Series EU1".
Limit	Integer	Limit the number of results per page that will be staged. This property is only available when the Data Source is set to "IoT Time Series EU1".
Asset ID	String	Specify the Asset ID. This property is only available when the Data Source property is set to "Asset ID" or "Variables".
Filter	JSON String	Provide a JSON string, which will act as an SQL filter for your query. This parameter is optional.
Location	Storage Location	Provide an S3 bucket path (AWS only), GCS bucket path (GCP only), or Azure Blob Storage path (Azure only) that will be used to store the data. Once on an S3 bucket, GCS bucket or Azure Blob, the data can be referenced by an external table. A folder will be created at this location with the same name as the Target Table.
Integration	Select	(GCP only) Choose your Google Cloud Storage Integration. Integrations are required to permit Snowflake to read data from and write to a Google Cloud Storage bucket. Integrations must be set up in advance of selecting them in Matillion ETL. To learn more about setting up a storage integration, read our Storage Integration Setup Guide.
Warehouse	Select	Choose a Snowflake warehouse that will run the load.
Database	Select	Select a database. A database is a logical grouping of schemas. Each database belongs to a single Snowflake account.
Schema	Select	Select the schema. A schema is a logical grouping of database "objects" (tables, views, etc.). Each schema belongs to a single database. The special value, [Environment Default], will use the schema defined in the environment. For more information on using multiple schemas, see this article.
Target Table	String	Specify the external table to be used. Warning: This table will be recreated and any existing table of the same name will be dropped.

Property	Setting	Description
Name	String	A human-readable name for the component. If a job includes more than one MindSphere Extract component, users might wish to name each component based on its data source.
API	Select	Select which MindSphere API to make queries to.
Data Source	Select	Select the data source. The available data sources depend on the chosen API. Upon selection, any properties specific to the chosen data source will be available for users to configure. This property is only available when the API property is set to "Asset Management EU1" or "Event Management EU1".
API Username	String	Input the API username for the chosen MindSphere app. For help locating your API Username, please refer to our MindSphere Extract Authentication Guide.
API Password	String	Input the API password for the chosen MindSphere app. For help locating your API password, please refer to our MindSphere Extract Authentication Guide. We recommend storing passwords in the Matillion ETL Password Manager.
App Name	String	Specify the name of target MindSphere application. Users can see an overview of their MindSphere apps, and create new apps, in the Developer Cockpit interface.
App Version	Version Number	Specify the version number of the target MindSphere application. Each application in the MindSphere Developer Cockpit interface has a version.
Host Tenant	String	Specify the Host Tenant. Users can find this credential in the MindSphere Asset Manager interface.
User Tenant	String	Specify the User Tenant. For help creating a user account or assigning MindSphere permissions, visit this page.
Entity Id	String	Specify your Entity Id. The Entity Id must exist in the IoT Entity Service. This property is only available when the Data Source is set to "IoT Time Series EU1".
Property Set Name	String	Specify the name of the Property Set. This property is only available when the Data Source is set to "IoT Time Series EU1".
Select	String	Specify the data to select from the Property Set. This property is only available when the Data Source is set to "IoT Time Series EU1".
Latest Value	Boolean	When True, Matillion ETL will only return the most recent value recorded for the metrics requested. This property is only available when the Data Source is set to "IoT Time Series EU1".
From	Timestamp	Specify the start point of the desired time range in the API call. The format is `YYYY-MM-DDThh:mm:ss.mmmZ` where `T` is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the Data Source is set to "IoT Time Series EU1".
To	Timestamp	Specify the end point of the desired time range in the API call. The format is `YYYY-MM-DDThh:mm:ss.mmmZ` where `T` is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the Data Source is set to "IoT Time Series EU1".
Limit	Integer	Limit the number of results per page that will be staged. This property is only available when the Data Source is set to "IoT Time Series EU1".
Asset ID	String	Specify the Asset ID. This property is only available when the Data Source property is set to "Asset ID" or "Variables".
Filter	JSON String	Provide a JSON string, which will act as an SQL filter for your query. This parameter is optional.
Page Limit	Integer	Limit the total number of pages to stage.
Location	Storage Location	Select an S3 bucket path that will be used to store the data. Once the data is on an S3 bucket, it can be referenced by an external table. A folder will be created at this location with the same name as the target table.
Type	Dropdown	Select between a standard table and an external table.
Standard Schema	Dropdown	Select the Redshift schema. The special value, [Environment Default], will use the schema defined in the Matillion ETL environment.
External Schema	Select	Select the table's external schema. To learn more about external schemas, please consult the "Configuring The Matillion ETL Client" section of the Getting Started With Amazon Redshift Spectrum documentation.
Target Table	String	Provide a name for the external table to be used. Warning: This table will be recreated and will drop any existing table of the same name.

Property	Setting	Description
Name	String	A human-readable name for the component. If a job includes more than one MindSphere Extract component, users might wish to name each component based on its data source.
API	Select	Select which MindSphere API to make queries to.
Data Source	Select	Select the data source. The available data sources depend on the chosen API. Upon selection, any properties specific to the chosen data source will be available for users to configure. This property is only available when the API property is set to "Asset Management EU1" or "Event Management EU1".
API Username	String	Input the API username for the chosen MindSphere app. For help locating your API Username, please refer to our MindSphere Extract Authentication Guide.
API Password	String	Input the API password for the chosen MindSphere app. For help locating your API password, please refer to our MindSphere Extract Authentication Guide. We recommend storing passwords in the Matillion ETL Password Manager.
App Name	String	Specify the name of target MindSphere application. Users can see an overview of their MindSphere apps, and create new apps, in the Developer Cockpit interface.
App Version	Version Number	Specify the version number of the target MindSphere application. Each application in the MindSphere Developer Cockpit interface has a version.
Host Tenant	String	Specify the Host Tenant. Users can find this credential in the MindSphere Asset Manager interface.
User Tenant	String	Specify the User Tenant. For help creating a user account or assigning MindSphere permissions, visit this page.
Entity Id	String	Specify your Entity Id. The Entity Id must exist in the IoT Entity Service. This property is only available when the Data Source is set to "IoT Time Series EU1".
Property Set Name	String	Specify the name of the Property Set. This property is only available when the Data Source is set to "IoT Time Series EU1".
Select	String	Specify the data to select from the Property Set. This property is only available when the Data Source is set to "IoT Time Series EU1".
Latest Value	Boolean	When True, Matillion ETL will only return the most recent value recorded for the metrics requested. This property is only available when the Data Source is set to "IoT Time Series EU1".
From	Timestamp	Specify the start point of the desired time range in the API call. The format is `YYYY-MM-DDThh:mm:ss.mmmZ` where `T` is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the Data Source is set to "IoT Time Series EU1".
To	Timestamp	Specify the end point of the desired time range in the API call. The format is `YYYY-MM-DDThh:mm:ss.mmmZ` where `T` is the literal letter T and indicates the start of the time element. All timestamps must be passed in the ISO8601 format in UTC. This property is only available when the Data Source is set to "IoT Time Series EU1".
Limit	Integer	Limit the number of results per page that will be staged. This property is only available when the Data Source is set to "IoT Time Series EU1".
Asset ID	String	Specify the Asset ID. This property is only available when the Data Source property is set to "Asset ID" or "Variables".
Filter	JSON String	Provide a JSON string, which will act as an SQL filter for your query. This parameter is optional.
Page Limit	Integer	Limit the total number of pages to stage.
Table Type	Select	Select whether the table is Native (by default in BigQuery) or an external table.
Project	Select	Select the Google Bigquery project. The special value, [Environment Default], will use the project defined in the environment. For more information, refer to the BigQuery documentation.
Dataset	Select	Select the Google Bigquery dataset to load data into. The special value, [Environment Default], will use the dataset defined in the environment. For more information, refer to the BigQuery documentation.
Target Table	String	A name for the table. Warning: This table will be recreated and will drop any existing table of the same name. Only available when the table type is Native.
New Target Table	String	A name for the new external table. Only available when the table type is External.
Cloud Storage Staging Area	Cloud Storage Bucket	Specify the target Google Cloud Storage bucket to be used for staging the queried data. Users can either: Input the URL string of the Cloud Storage bucket following the template provided: `gs://<bucket>/<path>` Navigate through the file structure to select the target bucket. Only available when the table type is Native.
Location	Cloud Storage Bucket	Specify the target Google Cloud Storage bucket to be used for staging the queried data. Users can either: Input the URL string of the Cloud Storage bucket following the template provided: `gs://<bucket>/<path>` Navigate through the file structure to select the target bucket. Only available when the table type is External.
Load Options	Multiple Select	Clean Cloud Storage Files: Destroy staged files on Cloud Storage after loading data. Default is On. Cloud Storage File Prefix: Give staged file names a prefix of your choice. The default setting is an empty field. Recreate Target Table: Choose whether the component recreates its target table before the data load. If Off, the component will use an existing table or create one if it does not exist. Default is On. Use Grid Variable: Check this checkbox to use a grid variable. This box is unchecked by default.

What's Next

MindSphere Extract Authentication Guide