steps

Creates, updates, deletes, gets or lists a steps resource.

Overview

Name	steps
Type	Resource
Id	aws.emr.steps

Fields

The following fields are returned by SELECT queries:

describe_step

Name	Datatype	Description
action_on_failure	string	The action to take when the cluster step fails. Possible values are TERMINATE_CLUSTER, CANCEL_AND_WAIT, and CONTINUE. TERMINATE_JOB_FLOW is provided for backward compatibility. We recommend using TERMINATE_CLUSTER instead. If a cluster's StepConcurrencyLevel is greater than 1, do not use AddJobFlowSteps to submit a step with this parameter set to CANCEL_AND_WAIT or TERMINATE_CLUSTER. The step is not submitted and the action fails with a message that the ActionOnFailure setting is not valid. If you change a cluster's StepConcurrencyLevel to be greater than 1 while a step is running, the ActionOnFailure parameter may not behave as you expect. In this case, for a step that fails with this parameter set to CANCEL_AND_WAIT, pending steps and the running step are not canceled; for a step that fails with this parameter set to TERMINATE_CLUSTER, the cluster does not terminate. (TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, CONTINUE)
config	object	The Hadoop job configuration of the cluster step.
encryption_key_arn	string	The KMS key ARN to encrypt the logs published to the given Amazon S3 destination.
execution_role_arn	string	The Amazon Resource Name (ARN) of the runtime role for a step on the cluster. The runtime role can be a cross-account IAM role. The runtime role ARN is a combination of account ID, role name, and role type using the following format: arn:partition:service:region:account:resource. For example, arn:aws:IAM::1234567890:role/ReadOnly is a correctly formatted runtime role ARN.
id	string	The identifier of the cluster step.
log_uri	string	The Amazon S3 destination URI for log publishing.
name	string	The name of the cluster step.
status	object	The current execution status details of the cluster step.

list_steps

Name	Datatype	Description
action_on_failure	string	The action to take when the cluster step fails. Possible values are TERMINATE_CLUSTER, CANCEL_AND_WAIT, and CONTINUE. TERMINATE_JOB_FLOW is available for backward compatibility. (TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, CONTINUE)
config	object	The Hadoop job configuration of the cluster step.
encryption_key_arn	string	The KMS key ARN to encrypt the logs published to the given Amazon S3 destination.
id	string	The identifier of the cluster step.
log_uri	string	The Amazon S3 destination URI for log publishing.
name	string	The name of the cluster step.
status	object	The current execution status details of the cluster step.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
describe_step	select	region		Provides more detail about the cluster step.
list_steps	select	region		Provides a list of steps for the cluster in reverse order unless you specify stepIds with the request or filter by StepStates. You can specify a maximum of 10 stepIDs. The CLI automatically paginates results to return a list greater than 50 steps. To return more than 50 steps using the CLI, specify a Marker, which is a pagination token that indicates the next set of steps to retrieve.
add_job_flow_steps	update	region, JobFlowId, Steps		AddJobFlowSteps adds new steps to a running cluster. A maximum of 256 steps are allowed in each job flow. If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using SSH to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. A step specifies the location of a JAR file stored either on the master node of the cluster or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step. Amazon EMR executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all Hadoop jobs started while the step was running must have completed and run successfully. You can only add steps to a cluster that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING. The string values passed into HadoopJarStep object cannot exceed a total of 10240 characters.
set_keep_job_flow_alive_when_no_steps	update	region, JobFlowIds, KeepJobFlowAliveWhenNoSteps		You can use the SetKeepJobFlowAliveWhenNoSteps to configure a cluster (job flow) to terminate after the step execution, i.e., all your steps are executed. If you want a transient cluster that shuts down after the last of the current executing steps are completed, you can configure SetKeepJobFlowAliveWhenNoSteps to false. If you want a long running cluster, configure SetKeepJobFlowAliveWhenNoSteps to true. For more information, see Managing Cluster Termination in the Amazon EMR Management Guide.
cancel_steps	exec	region, ClusterId, StepIds		Cancels a pending step or steps in a running cluster. Available only in Amazon EMR versions 4.8.0 and later, excluding version 5.0.0. A maximum of 256 steps are allowed in each CancelSteps request. CancelSteps is idempotent but asynchronous; it does not guarantee that a step will be canceled, even if the request is successfully submitted. When you use Amazon EMR releases 5.28.0 and later, you can cancel steps that are in a PENDING or RUNNING state. In earlier versions of Amazon EMR, you can only cancel steps that are in a PENDING state.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
region	string	AWS region (default: us-east-1)

SELECT examples

describe_step

Provides more detail about the cluster step.

SELECT
action_on_failure,
config,
encryption_key_arn,
execution_role_arn,
id,
log_uri,
name,
status
FROM aws.emr.steps
WHERE region = '{{ region }}' -- required
;

list_steps

Provides a list of steps for the cluster in reverse order unless you specify stepIds with the request or filter by StepStates. You can specify a maximum of 10 stepIDs. The CLI automatically paginates results to return a list greater than 50 steps. To return more than 50 steps using the CLI, specify a Marker, which is a pagination token that indicates the next set of steps to retrieve.

SELECT
action_on_failure,
config,
encryption_key_arn,
id,
log_uri,
name,
status
FROM aws.emr.steps
WHERE region = '{{ region }}' -- required
;

UPDATE examples

add_job_flow_steps

AddJobFlowSteps adds new steps to a running cluster. A maximum of 256 steps are allowed in each job flow. If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using SSH to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop. A step specifies the location of a JAR file stored either on the master node of the cluster or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step. Amazon EMR executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all Hadoop jobs started while the step was running must have completed and run successfully. You can only add steps to a cluster that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING. The string values passed into HadoopJarStep object cannot exceed a total of 10240 characters.

UPDATE aws.emr.steps
SET 
JobFlowId = '{{ JobFlowId }}',
Steps = '{{ Steps }}',
ExecutionRoleArn = '{{ ExecutionRoleArn }}'
WHERE 
region = '{{ region }}' --required
AND JobFlowId = '{{ JobFlowId }}' --required
AND Steps = '{{ Steps }}' --required
RETURNING
step_ids;

set_keep_job_flow_alive_when_no_steps

You can use the SetKeepJobFlowAliveWhenNoSteps to configure a cluster (job flow) to terminate after the step execution, i.e., all your steps are executed. If you want a transient cluster that shuts down after the last of the current executing steps are completed, you can configure SetKeepJobFlowAliveWhenNoSteps to false. If you want a long running cluster, configure SetKeepJobFlowAliveWhenNoSteps to true. For more information, see Managing Cluster Termination in the Amazon EMR Management Guide.

UPDATE aws.emr.steps
SET 
JobFlowIds = '{{ JobFlowIds }}',
KeepJobFlowAliveWhenNoSteps = {{ KeepJobFlowAliveWhenNoSteps }}
WHERE 
region = '{{ region }}' --required
AND JobFlowIds = '{{ JobFlowIds }}' --required
AND KeepJobFlowAliveWhenNoSteps = {{ KeepJobFlowAliveWhenNoSteps }} --required;

Lifecycle Methods

cancel_steps

Cancels a pending step or steps in a running cluster. Available only in Amazon EMR versions 4.8.0 and later, excluding version 5.0.0. A maximum of 256 steps are allowed in each CancelSteps request. CancelSteps is idempotent but asynchronous; it does not guarantee that a step will be canceled, even if the request is successfully submitted. When you use Amazon EMR releases 5.28.0 and later, you can cancel steps that are in a PENDING or RUNNING state. In earlier versions of Amazon EMR, you can only cancel steps that are in a PENDING state.

EXEC aws.emr.steps.cancel_steps 
@region='{{ region }}' --required 
@@json=
'{
"ClusterId": "{{ ClusterId }}", 
"StepIds": "{{ StepIds }}", 
"StepCancellationOption": "{{ StepCancellationOption }}"
}'
;