pipelines

Creates, updates, deletes, gets or lists a pipelines resource.

Overview

Name	pipelines
Type	Resource
Id	aws.sagemaker.pipelines

Fields

The following fields are returned by SELECT queries:

describe_pipeline

Name	Datatype	Description
created_by	object	Information about the user who created or modified a SageMaker resource.
creation_time	string (date-time)	The time when the pipeline was created.
last_modified_by	object	Information about the user who created or modified a SageMaker resource.
last_modified_time	string (date-time)	The time when the pipeline was last modified.
last_run_time	string (date-time)	The time when the pipeline was last run.
parallelism_configuration	object	Lists the parallelism configuration applied to the pipeline.
pipeline_arn	string	The Amazon Resource Name (ARN) of the pipeline. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:([0-9]{12}|aws):pipeline/.*</code>)
pipeline_definition	string	The JSON pipeline definition. (pattern: <code>.(?:[ \r\n\t].)*</code>)
pipeline_description	string	The description of the pipeline. (pattern: <code>.*</code>)
pipeline_display_name	string	The display name of the pipeline. (pattern: <code>[a-zA-Z0-9](-*[a-zA-Z0-9]){0,255}</code>)
pipeline_name	string	The name of the pipeline. (pattern: <code>[a-zA-Z0-9](-*[a-zA-Z0-9]){0,255}</code>)
pipeline_status	string	The status of the pipeline execution. (Active, Deleting)
pipeline_version_description	string	The description of the pipeline version. (pattern: <code>.*</code>)
pipeline_version_display_name	string	The display name of the pipeline version. (pattern: <code>[a-zA-Z0-9](-*[a-zA-Z0-9]){0,81}</code>)
role_arn	string	The Amazon Resource Name (ARN) that the pipeline uses to execute. (pattern: <code>arn:aws[a-z-]*:iam::\d{12}:role/?[a-zA-Z_0-9+=,.@-_/]+</code>)

list_pipelines

Name	Datatype	Description
creation_time	string (date-time)	The creation time of the pipeline.
last_execution_time	string (date-time)	The last time that a pipeline execution began.
last_modified_time	string (date-time)	The time that the pipeline was last modified.
pipeline_arn	string	The Amazon Resource Name (ARN) of the pipeline. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:([0-9]{12}|aws):pipeline/.*</code>)
pipeline_description	string	The description of the pipeline. (pattern: <code>.*</code>)
pipeline_display_name	string	The display name of the pipeline. (pattern: <code>[a-zA-Z0-9](-*[a-zA-Z0-9]){0,255}</code>)
pipeline_name	string	The name of the pipeline. (pattern: <code>[a-zA-Z0-9](-*[a-zA-Z0-9]){0,255}</code>)
role_arn	string	The Amazon Resource Name (ARN) that the pipeline used to execute. (pattern: <code>arn:aws[a-z-]*:iam::\d{12}:role/?[a-zA-Z_0-9+=,.@-_/]+</code>)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
describe_pipeline	select	region		Describes the details of a pipeline.
list_pipelines	select	region		Gets a list of pipelines.
create_pipeline	insert	region, PipelineName, ClientRequestToken, RoleArn		Creates a pipeline using a JSON pipeline definition.
update_pipeline	update	region, PipelineName		Updates a pipeline.
delete_pipeline	delete	region		Deletes a pipeline if there are no running instances of the pipeline. To delete a pipeline, you must stop all running instances of the pipeline using the StopPipelineExecution API. When you delete a pipeline, all instances of the pipeline are deleted.
start_pipeline_execution	exec	region, PipelineName, ClientRequestToken		Starts a pipeline execution.
stop_pipeline_execution	exec	region, PipelineExecutionArn, ClientRequestToken		Stops a pipeline execution. Callback Step A pipeline execution won't stop while a callback step is running. When you call StopPipelineExecution on a pipeline execution with a running callback step, SageMaker Pipelines sends an additional Amazon SQS message to the specified SQS queue. The body of the SQS message contains a "Status" field which is set to "Stopping". You should add logic to your Amazon SQS message consumer to take any needed action (for example, resource cleanup) upon receipt of the message followed by a call to SendPipelineExecutionStepSuccess or SendPipelineExecutionStepFailure. Only when SageMaker Pipelines receives one of these calls will it stop the pipeline execution. Lambda Step A pipeline execution can't be stopped while a lambda step is running because the Lambda function invoked by the lambda step can't be stopped. If you attempt to stop the execution while the Lambda function is running, the pipeline waits for the Lambda function to finish or until the timeout is hit, whichever occurs first, and then stops. If the Lambda function finishes, the pipeline execution status is Stopped. If the timeout is hit the pipeline execution status is Failed.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
region	string	AWS region (default: us-east-1)

SELECT examples

describe_pipeline

Describes the details of a pipeline.

SELECT
created_by,
creation_time,
last_modified_by,
last_modified_time,
last_run_time,
parallelism_configuration,
pipeline_arn,
pipeline_definition,
pipeline_description,
pipeline_display_name,
pipeline_name,
pipeline_status,
pipeline_version_description,
pipeline_version_display_name,
role_arn
FROM aws.sagemaker.pipelines
WHERE region = '{{ region }}' -- required
;

list_pipelines

Gets a list of pipelines.

SELECT
creation_time,
last_execution_time,
last_modified_time,
pipeline_arn,
pipeline_description,
pipeline_display_name,
pipeline_name,
role_arn
FROM aws.sagemaker.pipelines
WHERE region = '{{ region }}' -- required
;

INSERT examples

create_pipeline

Creates a pipeline using a JSON pipeline definition.

INSERT INTO aws.sagemaker.pipelines (
PipelineName,
PipelineDisplayName,
PipelineDefinition,
PipelineDefinitionS3Location,
PipelineDescription,
ClientRequestToken,
RoleArn,
Tags,
ParallelismConfiguration,
region
)
SELECT 
'{{ PipelineName }}' /* required */,
'{{ PipelineDisplayName }}',
'{{ PipelineDefinition }}',
'{{ PipelineDefinitionS3Location }}',
'{{ PipelineDescription }}',
'{{ ClientRequestToken }}' /* required */,
'{{ RoleArn }}' /* required */,
'{{ Tags }}',
'{{ ParallelismConfiguration }}',
'{{ region }}'
RETURNING
pipeline_arn
;

Manifest

# Description fields are for documentation purposes
- name: pipelines
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the pipelines resource.
  - name: PipelineName
    value: "{{ PipelineName }}"
    description: |
      The name of the pipeline.
  - name: PipelineDisplayName
    value: "{{ PipelineDisplayName }}"
    description: |
      The display name of the pipeline.
  - name: PipelineDefinition
    value: "{{ PipelineDefinition }}"
    description: |
      The JSON pipeline definition of the pipeline.
  - name: PipelineDefinitionS3Location
    description: |
      The location of the pipeline definition stored in Amazon S3. If specified, SageMaker will retrieve the pipeline definition from this location.
    value:
      Bucket: "{{ Bucket }}"
      ObjectKey: "{{ ObjectKey }}"
      VersionId: "{{ VersionId }}"
  - name: PipelineDescription
    value: "{{ PipelineDescription }}"
    description: |
      A description of the pipeline.
  - name: ClientRequestToken
    value: "{{ ClientRequestToken }}"
    description: |
      A unique, case-sensitive identifier that you provide to ensure the idempotency of the operation. An idempotent operation completes no more than one time.
  - name: RoleArn
    value: "{{ RoleArn }}"
    description: |
      The Amazon Resource Name (ARN) of the role used by the pipeline to access and create resources.
  - name: Tags
    description: |
      A list of tags to apply to the created pipeline.
    value:
      - Key: "{{ Key }}"
        Value: "{{ Value }}"
  - name: ParallelismConfiguration
    description: |
      This is the configuration that controls the parallelism of the pipeline. If specified, it applies to all runs of this pipeline by default.
    value:
      MaxParallelExecutionSteps: {{ MaxParallelExecutionSteps }}

UPDATE examples

update_pipeline

Updates a pipeline.

UPDATE aws.sagemaker.pipelines
SET 
PipelineName = '{{ PipelineName }}',
PipelineDisplayName = '{{ PipelineDisplayName }}',
PipelineDefinition = '{{ PipelineDefinition }}',
PipelineDefinitionS3Location = '{{ PipelineDefinitionS3Location }}',
PipelineDescription = '{{ PipelineDescription }}',
RoleArn = '{{ RoleArn }}',
ParallelismConfiguration = '{{ ParallelismConfiguration }}'
WHERE 
region = '{{ region }}' --required
AND PipelineName = '{{ PipelineName }}' --required
RETURNING
pipeline_arn,
pipeline_version_id;

DELETE examples

delete_pipeline

Deletes a pipeline if there are no running instances of the pipeline. To delete a pipeline, you must stop all running instances of the pipeline using the StopPipelineExecution API. When you delete a pipeline, all instances of the pipeline are deleted.

DELETE FROM aws.sagemaker.pipelines
WHERE region = '{{ region }}' --required
;

Lifecycle Methods

start_pipeline_execution

Starts a pipeline execution.

EXEC aws.sagemaker.pipelines.start_pipeline_execution 
@region='{{ region }}' --required 
@@json=
'{
"PipelineName": "{{ PipelineName }}", 
"PipelineExecutionDisplayName": "{{ PipelineExecutionDisplayName }}", 
"PipelineParameters": "{{ PipelineParameters }}", 
"PipelineExecutionDescription": "{{ PipelineExecutionDescription }}", 
"ClientRequestToken": "{{ ClientRequestToken }}", 
"ParallelismConfiguration": "{{ ParallelismConfiguration }}", 
"SelectiveExecutionConfig": "{{ SelectiveExecutionConfig }}", 
"PipelineVersionId": {{ PipelineVersionId }}, 
"MlflowExperimentName": "{{ MlflowExperimentName }}"
}'
;

stop_pipeline_execution

Stops a pipeline execution. Callback Step A pipeline execution won't stop while a callback step is running. When you call StopPipelineExecution on a pipeline execution with a running callback step, SageMaker Pipelines sends an additional Amazon SQS message to the specified SQS queue. The body of the SQS message contains a "Status" field which is set to "Stopping". You should add logic to your Amazon SQS message consumer to take any needed action (for example, resource cleanup) upon receipt of the message followed by a call to SendPipelineExecutionStepSuccess or SendPipelineExecutionStepFailure. Only when SageMaker Pipelines receives one of these calls will it stop the pipeline execution. Lambda Step A pipeline execution can't be stopped while a lambda step is running because the Lambda function invoked by the lambda step can't be stopped. If you attempt to stop the execution while the Lambda function is running, the pipeline waits for the Lambda function to finish or until the timeout is hit, whichever occurs first, and then stops. If the Lambda function finishes, the pipeline execution status is Stopped. If the timeout is hit the pipeline execution status is Failed.

EXEC aws.sagemaker.pipelines.stop_pipeline_execution 
@region='{{ region }}' --required 
@@json=
'{
"PipelineExecutionArn": "{{ PipelineExecutionArn }}", 
"ClientRequestToken": "{{ ClientRequestToken }}"
}'
;