data_integration_flows

Creates, updates, deletes, gets or lists a data_integration_flows resource.

Overview

Name	data_integration_flows
Type	Resource
Id	aws.supplychain.data_integration_flows

Fields

The following fields are returned by SELECT queries:

get_data_integration_flow

Name	Datatype	Description
name	string	The DataIntegrationFlow name. (pattern: <code>[A-Za-z0-9-]+</code>)
created_time	string (date-time)	The DataIntegrationFlow creation timestamp.
instance_id	string	The DataIntegrationFlow instance ID. (pattern: <code>[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}</code>)
last_modified_time	string (date-time)	The DataIntegrationFlow last modified timestamp.
sources	array	The DataIntegrationFlow source configurations.
target	object	The DataIntegrationFlow target parameters.
transformation	object	The DataIntegrationFlow transformation parameters.

list_data_integration_flows

Name	Datatype	Description
name	string	The DataIntegrationFlow name. (pattern: <code>[A-Za-z0-9-]+</code>)
created_time	string (date-time)	The DataIntegrationFlow creation timestamp.
instance_id	string	The DataIntegrationFlow instance ID. (pattern: <code>[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}</code>)
last_modified_time	string (date-time)	The DataIntegrationFlow last modified timestamp.
sources	array	The DataIntegrationFlow source configurations.
target	object	The DataIntegrationFlow target parameters.
transformation	object	The DataIntegrationFlow transformation parameters.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_data_integration_flow	select	instance_id, name, region		Enables you to programmatically view a specific data pipeline for the provided Amazon Web Services Supply Chain instance and DataIntegrationFlow name.
list_data_integration_flows	select	instance_id, region	nextToken, maxResults	Enables you to programmatically list all data pipelines for the provided Amazon Web Services Supply Chain instance.
create_data_integration_flow	insert	instance_id, name, region, sources, transformation, target		Enables you to programmatically create a data pipeline to ingest data from source systems such as Amazon S3 buckets, to a predefined Amazon Web Services Supply Chain dataset (product, inbound_order) or a temporary dataset along with the data transformation query provided with the API.
update_data_integration_flow	update	instance_id, name, region		Enables you to programmatically update an existing data pipeline to ingest data from the source systems such as, Amazon S3 buckets, to a predefined Amazon Web Services Supply Chain dataset (product, inbound_order) or a temporary dataset along with the data transformation query provided with the API.
delete_data_integration_flow	delete	instance_id, name, region		Enable you to programmatically delete an existing data pipeline for the provided Amazon Web Services Supply Chain instance and DataIntegrationFlow name.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
instance_id	string	The Amazon Web Services Supply Chain instance identifier.
name	string	The name of the DataIntegrationFlow to be deleted.
region	string	AWS region (default: us-east-1)
maxResults	integer	Specify the maximum number of DataIntegrationFlows to fetch in one paginated request.
nextToken	string	The pagination token to fetch the next page of the DataIntegrationFlows.

SELECT examples

get_data_integration_flow

Enables you to programmatically view a specific data pipeline for the provided Amazon Web Services Supply Chain instance and DataIntegrationFlow name.

SELECT
name,
created_time,
instance_id,
last_modified_time,
sources,
target,
transformation
FROM aws.supplychain.data_integration_flows
WHERE instance_id = '{{ instance_id }}' -- required
AND name = '{{ name }}' -- required
AND region = '{{ region }}' -- required
;

list_data_integration_flows

Enables you to programmatically list all data pipelines for the provided Amazon Web Services Supply Chain instance.

SELECT
name,
created_time,
instance_id,
last_modified_time,
sources,
target,
transformation
FROM aws.supplychain.data_integration_flows
WHERE instance_id = '{{ instance_id }}' -- required
AND region = '{{ region }}' -- required
AND nextToken = '{{ nextToken }}'
AND maxResults = '{{ maxResults }}'
;

INSERT examples

create_data_integration_flow

Enables you to programmatically create a data pipeline to ingest data from source systems such as Amazon S3 buckets, to a predefined Amazon Web Services Supply Chain dataset (product, inbound_order) or a temporary dataset along with the data transformation query provided with the API.

INSERT INTO aws.supplychain.data_integration_flows (
sources,
transformation,
target,
tags,
instance_id,
name,
region
)
SELECT 
'{{ sources }}' /* required */,
'{{ transformation }}' /* required */,
'{{ target }}' /* required */,
'{{ tags }}',
'{{ instance_id }}',
'{{ name }}',
'{{ region }}'
RETURNING
name,
instance_id
;

Manifest

# Description fields are for documentation purposes
- name: data_integration_flows
props:
  - name: instance_id
    value: "{{ instance_id }}"
    description: Required parameter for the data_integration_flows resource.
  - name: name
    value: "{{ name }}"
    description: Required parameter for the data_integration_flows resource.
  - name: region
    value: "{{ region }}"
    description: Required parameter for the data_integration_flows resource.
  - name: sources
    value:
      - sourceType: "{{ sourceType }}"
        sourceName: "{{ sourceName }}"
        s3Source:
          bucketName: "{{ bucketName }}"
          prefix: "{{ prefix }}"
          options:
            fileType: "{{ fileType }}"
        datasetSource:
          datasetIdentifier: "{{ datasetIdentifier }}"
          options:
            loadType: "{{ loadType }}"
            dedupeRecords: {{ dedupeRecords }}
            dedupeStrategy:
              type_: "{{ type_ }}"
              fieldPriority:
                fields: "{{ fields }}"
  - name: transformation
    description: |
      The DataIntegrationFlow transformation parameters.
    value:
      transformationType: "{{ transformationType }}"
      sqlTransformation:
        query: "{{ query }}"
  - name: target
    description: |
      The DataIntegrationFlow target parameters.
    value:
      targetType: "{{ targetType }}"
      s3Target:
        bucketName: "{{ bucketName }}"
        prefix: "{{ prefix }}"
        options:
          fileType: "{{ fileType }}"
      datasetTarget:
        datasetIdentifier: "{{ datasetIdentifier }}"
        options:
          loadType: "{{ loadType }}"
          dedupeRecords: {{ dedupeRecords }}
          dedupeStrategy:
            type_: "{{ type_ }}"
            fieldPriority:
              fields: "{{ fields }}"
  - name: tags
    value: "{{ tags }}"

UPDATE examples

update_data_integration_flow

Enables you to programmatically update an existing data pipeline to ingest data from the source systems such as, Amazon S3 buckets, to a predefined Amazon Web Services Supply Chain dataset (product, inbound_order) or a temporary dataset along with the data transformation query provided with the API.

UPDATE aws.supplychain.data_integration_flows
SET 
sources = '{{ sources }}',
transformation = '{{ transformation }}',
target = '{{ target }}'
WHERE 
instance_id = '{{ instance_id }}' --required
AND name = '{{ name }}' --required
AND region = '{{ region }}' --required
RETURNING
flow;

DELETE examples

delete_data_integration_flow

Enable you to programmatically delete an existing data pipeline for the provided Amazon Web Services Supply Chain instance and DataIntegrationFlow name.

DELETE FROM aws.supplychain.data_integration_flows
WHERE instance_id = '{{ instance_id }}' --required
AND name = '{{ name }}' --required
AND region = '{{ region }}' --required
;