pipelines

Creates, updates, deletes, gets or lists a pipelines resource.

Overview

Name	pipelines
Type	Resource
Id	aws.datapipeline.pipelines

Fields

The following fields are returned by SELECT queries:

describe_pipelines

Name	Datatype	Description
pipeline_description_list	array	An array of descriptions for the specified pipelines.

list_pipelines

Name	Datatype	Description
id	string	The ID of the pipeline that was assigned by AWS Data Pipeline. This is a string of the form df-297EG78HU43EEXAMPLE. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>)
name	string	The name of the pipeline. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
describe_pipelines	select	region		Retrieves metadata about one or more pipelines. The information retrieved includes the name of the pipeline, the pipeline identifier, its current state, and the user account that owns the pipeline. Using account credentials, you can retrieve metadata about pipelines that you or your IAM users have created. If you are using an IAM user account, you can retrieve metadata about only those pipelines for which you have read permissions. To retrieve the full pipeline definition instead of metadata about the pipeline, call GetPipelineDefinition.
list_pipelines	select	region		Lists the pipeline identifiers for all active pipelines that you have permission to access.
create_pipeline	insert	region, name, uniqueId		Creates a new, empty pipeline. Use PutPipelineDefinition to populate the pipeline.
set_status	update	region, pipelineId, objectIds, status		Requests that the status of the specified physical or logical pipeline objects be updated in the specified pipeline. This update might not occur immediately, but is eventually consistent. The status that can be set depends on the type of object (for example, DataNode or Activity). You cannot perform this operation on FINISHED pipelines and attempting to do so returns InvalidRequestException.
delete_pipeline	delete	region		Deletes a pipeline, its pipeline definition, and its run history. AWS Data Pipeline attempts to cancel instances associated with the pipeline that are currently being processed by task runners. Deleting a pipeline cannot be undone. You cannot query or restore a deleted pipeline. To temporarily pause a pipeline instead of deleting it, call SetStatus with the status set to PAUSE on individual components. Components that are paused by SetStatus can be resumed.
activate_pipeline	exec	region, pipelineId		Validates the specified pipeline and starts processing pipeline tasks. If the pipeline does not pass validation, activation fails. If you need to pause the pipeline to investigate an issue with a component, such as a data source or script, call DeactivatePipeline. To activate a finished pipeline, modify the end date for the pipeline and then activate it.
deactivate_pipeline	exec	region, pipelineId		Deactivates the specified running pipeline. The pipeline is set to the DEACTIVATING state until the deactivation process completes. To resume a deactivated pipeline, use ActivatePipeline. By default, the pipeline resumes from the last completed execution. Optionally, you can specify the date and time to resume the pipeline.
evaluate_expression	exec	region, pipelineId, objectId, expression		Task runners call EvaluateExpression to evaluate a string in the context of the specified object. For example, a task runner can evaluate SQL queries stored in Amazon S3.
validate_pipeline_definition	exec	region, pipelineId, pipelineObjects		Validates the specified pipeline definition to ensure that it is well formed and can be run without error.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
region	string	AWS region (default: us-east-1)

SELECT examples

describe_pipelines

Retrieves metadata about one or more pipelines. The information retrieved includes the name of the pipeline, the pipeline identifier, its current state, and the user account that owns the pipeline. Using account credentials, you can retrieve metadata about pipelines that you or your IAM users have created. If you are using an IAM user account, you can retrieve metadata about only those pipelines for which you have read permissions. To retrieve the full pipeline definition instead of metadata about the pipeline, call GetPipelineDefinition.

SELECT
pipeline_description_list
FROM aws.datapipeline.pipelines
WHERE region = '{{ region }}' -- required
;

list_pipelines

Lists the pipeline identifiers for all active pipelines that you have permission to access.

SELECT
id,
name
FROM aws.datapipeline.pipelines
WHERE region = '{{ region }}' -- required
;

INSERT examples

create_pipeline

Creates a new, empty pipeline. Use PutPipelineDefinition to populate the pipeline.

INSERT INTO aws.datapipeline.pipelines (
name,
uniqueId,
description,
tags,
region
)
SELECT 
'{{ name }}' /* required */,
'{{ uniqueId }}' /* required */,
'{{ description }}',
'{{ tags }}',
'{{ region }}'
RETURNING
pipeline_id
;

Manifest

# Description fields are for documentation purposes
- name: pipelines
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the pipelines resource.
  - name: name
    value: "{{ name }}"
    description: |
      The name for the pipeline. You can use the same name for multiple pipelines associated with your AWS account, because AWS Data Pipeline assigns each pipeline a unique pipeline identifier.
  - name: uniqueId
    value: "{{ uniqueId }}"
    description: |
      A unique identifier. This identifier is not the same as the pipeline identifier assigned by AWS Data Pipeline. You are responsible for defining the format and ensuring the uniqueness of this identifier. You use this parameter to ensure idempotency during repeated calls to CreatePipeline. For example, if the first call to CreatePipeline does not succeed, you can pass in the same unique identifier and pipeline name combination on a subsequent call to CreatePipeline. CreatePipeline ensures that if a pipeline already exists with the same name and unique identifier, a new pipeline is not created. Instead, you'll receive the pipeline identifier from the previous attempt. The uniqueness of the name and unique identifier combination is scoped to the AWS account or IAM user credentials.
  - name: description
    value: "{{ description }}"
    description: |
      The description for the pipeline.
  - name: tags
    description: |
      A list of tags to associate with the pipeline at creation. Tags let you control access to pipelines. For more information, see Controlling User Access to Pipelines in the AWS Data Pipeline Developer Guide.
    value:
      - key: "{{ key }}"
        value: "{{ value }}"

UPDATE examples

set_status

Requests that the status of the specified physical or logical pipeline objects be updated in the specified pipeline. This update might not occur immediately, but is eventually consistent. The status that can be set depends on the type of object (for example, DataNode or Activity). You cannot perform this operation on FINISHED pipelines and attempting to do so returns InvalidRequestException.

UPDATE aws.datapipeline.pipelines
SET 
pipelineId = '{{ pipelineId }}',
objectIds = '{{ objectIds }}',
status = '{{ status }}'
WHERE 
region = '{{ region }}' --required
AND pipelineId = '{{ pipelineId }}' --required
AND objectIds = '{{ objectIds }}' --required
AND status = '{{ status }}' --required;

DELETE examples

delete_pipeline

Deletes a pipeline, its pipeline definition, and its run history. AWS Data Pipeline attempts to cancel instances associated with the pipeline that are currently being processed by task runners. Deleting a pipeline cannot be undone. You cannot query or restore a deleted pipeline. To temporarily pause a pipeline instead of deleting it, call SetStatus with the status set to PAUSE on individual components. Components that are paused by SetStatus can be resumed.

DELETE FROM aws.datapipeline.pipelines
WHERE region = '{{ region }}' --required
;

Lifecycle Methods

activate_pipeline

Validates the specified pipeline and starts processing pipeline tasks. If the pipeline does not pass validation, activation fails. If you need to pause the pipeline to investigate an issue with a component, such as a data source or script, call DeactivatePipeline. To activate a finished pipeline, modify the end date for the pipeline and then activate it.

EXEC aws.datapipeline.pipelines.activate_pipeline 
@region='{{ region }}' --required 
@@json=
'{
"pipelineId": "{{ pipelineId }}", 
"parameterValues": "{{ parameterValues }}", 
"startTimestamp": "{{ startTimestamp }}"
}'
;

deactivate_pipeline

Deactivates the specified running pipeline. The pipeline is set to the DEACTIVATING state until the deactivation process completes. To resume a deactivated pipeline, use ActivatePipeline. By default, the pipeline resumes from the last completed execution. Optionally, you can specify the date and time to resume the pipeline.

EXEC aws.datapipeline.pipelines.deactivate_pipeline 
@region='{{ region }}' --required 
@@json=
'{
"pipelineId": "{{ pipelineId }}", 
"cancelActive": {{ cancelActive }}
}'
;

evaluate_expression

Task runners call EvaluateExpression to evaluate a string in the context of the specified object. For example, a task runner can evaluate SQL queries stored in Amazon S3.

EXEC aws.datapipeline.pipelines.evaluate_expression 
@region='{{ region }}' --required 
@@json=
'{
"pipelineId": "{{ pipelineId }}", 
"objectId": "{{ objectId }}", 
"expression": "{{ expression }}"
}'
;

validate_pipeline_definition

Validates the specified pipeline definition to ensure that it is well formed and can be run without error.

EXEC aws.datapipeline.pipelines.validate_pipeline_definition 
@region='{{ region }}' --required 
@@json=
'{
"pipelineId": "{{ pipelineId }}", 
"pipelineObjects": "{{ pipelineObjects }}", 
"parameterObjects": "{{ parameterObjects }}", 
"parameterValues": "{{ parameterValues }}"
}'
;