pipelines

Creates, updates, deletes, gets or lists a pipelines resource.

Overview

Name	pipelines
Type	Resource
Id	aws.osis.pipelines

Fields

The following fields are returned by SELECT queries:

get_pipeline

Name	Datatype	Description
buffer_options	object	Options that specify the configuration of a persistent buffer. To configure how OpenSearch Ingestion encrypts this data, set the EncryptionAtRestOptions. For more information, see Persistent buffering.
created_at	string (date-time)	The date and time when the pipeline was created.
destinations	array	Destinations to which the pipeline writes data.
encryption_at_rest_options	object	Options to control how OpenSearch encrypts buffer data.
ingest_endpoint_urls	array	The ingestion endpoints for the pipeline, which you can send data to.
last_updated_at	string (date-time)	The date and time when the pipeline was last updated.
log_publishing_options	object	Container for the values required to configure logging for the pipeline. If you don't specify these values, OpenSearch Ingestion will not publish logs from your application to CloudWatch Logs.
max_units	integer	The maximum pipeline capacity, in Ingestion Compute Units (ICUs).
min_units	integer	The minimum pipeline capacity, in Ingestion Compute Units (ICUs).
pipeline_arn	string	The Amazon Resource Name (ARN) of the pipeline.
pipeline_configuration_body	string	The Data Prepper pipeline configuration in YAML format.
pipeline_name	string	The name of the pipeline.
pipeline_role_arn	string	The Amazon Resource Name (ARN) of the IAM role that the pipeline uses to access AWS resources. (pattern: <code>^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b|aws-iso-e|aws-iso-f):iam::[0-9]+:role/.*$</code>)
service_vpc_endpoints	array	A list of VPC endpoints that OpenSearch Ingestion has created to other Amazon Web Services services.
status	string	The current status of the pipeline. (CREATING, ACTIVE, UPDATING, DELETING, CREATE_FAILED, UPDATE_FAILED, STARTING, START_FAILED, STOPPING, STOPPED)
status_reason	object	The reason for the current status of the pipeline.
tags	array	A list of tags associated with the given pipeline.
vpc_endpoint_service	string	The VPC endpoint service name for the pipeline.
vpc_endpoints	array	The VPC interface endpoints that have access to the pipeline.

list_pipelines

Name	Datatype	Description
next_token	string	When nextToken is returned, there are more results available. The value of nextToken is a unique pagination token for each page. Make the call again using the returned token to retrieve the next page. (pattern: <code>^([\s\S]*)$</code>)
pipelines	array	A list of all existing Data Prepper pipelines.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_pipeline	select	pipeline_name, region		Retrieves information about an OpenSearch Ingestion pipeline.
list_pipelines	select	region	maxResults, nextToken	Lists all OpenSearch Ingestion pipelines in the current Amazon Web Services account and Region. For more information, see Viewing Amazon OpenSearch Ingestion pipelines.
create_pipeline	insert	region, PipelineName, MinUnits, MaxUnits, PipelineConfigurationBody		Creates an OpenSearch Ingestion pipeline. For more information, see Creating Amazon OpenSearch Ingestion pipelines.
update_pipeline	update	pipeline_name, region		Updates an OpenSearch Ingestion pipeline. For more information, see Updating Amazon OpenSearch Ingestion pipelines.
delete_pipeline	delete	pipeline_name, region		Deletes an OpenSearch Ingestion pipeline. For more information, see Deleting Amazon OpenSearch Ingestion pipelines.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
pipeline_name	string	The name of the pipeline to delete.
region	string	AWS region (default: us-east-1)
maxResults	integer	An optional parameter that specifies the maximum number of results to return. You can use nextToken to get the next page of results.
nextToken	string	If your initial ListPipelines operation returns a nextToken, you can include the returned nextToken in subsequent ListPipelines operations, which returns results in the next page.

SELECT examples

get_pipeline

Retrieves information about an OpenSearch Ingestion pipeline.

SELECT
buffer_options,
created_at,
destinations,
encryption_at_rest_options,
ingest_endpoint_urls,
last_updated_at,
log_publishing_options,
max_units,
min_units,
pipeline_arn,
pipeline_configuration_body,
pipeline_name,
pipeline_role_arn,
service_vpc_endpoints,
status,
status_reason,
tags,
vpc_endpoint_service,
vpc_endpoints
FROM aws.osis.pipelines
WHERE pipeline_name = '{{ pipeline_name }}' -- required
AND region = '{{ region }}' -- required
;

list_pipelines

Lists all OpenSearch Ingestion pipelines in the current Amazon Web Services account and Region. For more information, see Viewing Amazon OpenSearch Ingestion pipelines.

SELECT
next_token,
pipelines
FROM aws.osis.pipelines
WHERE region = '{{ region }}' -- required
AND maxResults = '{{ maxResults }}'
AND nextToken = '{{ nextToken }}'
;

INSERT examples

create_pipeline

Creates an OpenSearch Ingestion pipeline. For more information, see Creating Amazon OpenSearch Ingestion pipelines.

INSERT INTO aws.osis.pipelines (
PipelineName,
MinUnits,
MaxUnits,
PipelineConfigurationBody,
LogPublishingOptions,
VpcOptions,
BufferOptions,
EncryptionAtRestOptions,
Tags,
PipelineRoleArn,
region
)
SELECT 
'{{ PipelineName }}' /* required */,
{{ MinUnits }} /* required */,
{{ MaxUnits }} /* required */,
'{{ PipelineConfigurationBody }}' /* required */,
'{{ LogPublishingOptions }}',
'{{ VpcOptions }}',
'{{ BufferOptions }}',
'{{ EncryptionAtRestOptions }}',
'{{ Tags }}',
'{{ PipelineRoleArn }}',
'{{ region }}'
RETURNING
pipeline
;

Manifest

# Description fields are for documentation purposes
- name: pipelines
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the pipelines resource.
  - name: PipelineName
    value: "{{ PipelineName }}"
  - name: MinUnits
    value: {{ MinUnits }}
  - name: MaxUnits
    value: {{ MaxUnits }}
  - name: PipelineConfigurationBody
    value: "{{ PipelineConfigurationBody }}"
  - name: LogPublishingOptions
    description: |
      Container for the values required to configure logging for the pipeline. If you don't specify these values, OpenSearch Ingestion will not publish logs from your application to CloudWatch Logs.
    value:
      IsLoggingEnabled: {{ IsLoggingEnabled }}
      CloudWatchLogDestination:
        LogGroup: "{{ LogGroup }}"
  - name: VpcOptions
    description: |
      Options that specify the subnets and security groups for an OpenSearch Ingestion VPC endpoint.
    value:
      SubnetIds:
        - "{{ SubnetIds }}"
      SecurityGroupIds:
        - "{{ SecurityGroupIds }}"
      VpcAttachmentOptions:
        AttachToVpc: {{ AttachToVpc }}
        CidrBlock: "{{ CidrBlock }}"
      VpcEndpointManagement: "{{ VpcEndpointManagement }}"
  - name: BufferOptions
    description: |
      Options that specify the configuration of a persistent buffer. To configure how OpenSearch Ingestion encrypts this data, set the EncryptionAtRestOptions. For more information, see Persistent buffering.
    value:
      PersistentBufferEnabled: {{ PersistentBufferEnabled }}
  - name: EncryptionAtRestOptions
    description: |
      Options to control how OpenSearch encrypts buffer data.
    value:
      KmsKeyArn: "{{ KmsKeyArn }}"
  - name: Tags
    value:
      - Key: "{{ Key }}"
        Value: "{{ Value }}"
  - name: PipelineRoleArn
    value: "{{ PipelineRoleArn }}"

UPDATE examples

update_pipeline

Updates an OpenSearch Ingestion pipeline. For more information, see Updating Amazon OpenSearch Ingestion pipelines.

UPDATE aws.osis.pipelines
SET 
MinUnits = {{ MinUnits }},
MaxUnits = {{ MaxUnits }},
PipelineConfigurationBody = '{{ PipelineConfigurationBody }}',
LogPublishingOptions = '{{ LogPublishingOptions }}',
BufferOptions = '{{ BufferOptions }}',
EncryptionAtRestOptions = '{{ EncryptionAtRestOptions }}',
PipelineRoleArn = '{{ PipelineRoleArn }}'
WHERE 
pipeline_name = '{{ pipeline_name }}' --required
AND region = '{{ region }}' --required
RETURNING
pipeline;

DELETE examples

delete_pipeline

Deletes an OpenSearch Ingestion pipeline. For more information, see Deleting Amazon OpenSearch Ingestion pipelines.

DELETE FROM aws.osis.pipelines
WHERE pipeline_name = '{{ pipeline_name }}' --required
AND region = '{{ region }}' --required
;