clusters
Creates, updates, deletes, gets or lists a clusters resource.
Overview
| Name | clusters |
| Type | Resource |
| Id | aws.emr.clusters |
Fields
The following fields are returned by SELECT queries:
- describe_cluster
- list_clusters
| Name | Datatype | Description |
|---|---|---|
applications | array | The applications installed on this cluster. |
auto_scaling_role | string | An IAM role for automatic scaling policies. The default role is EMR_AutoScaling_DefaultRole. The IAM role provides permissions that the automatic scaling feature requires to launch and terminate Amazon EC2 instances in an instance group. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>) |
auto_terminate | boolean | Specifies whether the cluster should terminate after completing all steps. |
cluster_arn | string | The Amazon Resource Name of the cluster. |
configurations | array | Applies only to Amazon EMR releases 4.x and later. The list of configurations that are supplied to the Amazon EMR cluster. |
custom_ami_id | string | Available only in Amazon EMR releases 5.7.0 and later. The ID of a custom Amazon EBS-backed Linux AMI if the cluster uses a custom AMI. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>) |
ebs_root_volume_iops | integer | The IOPS, of the Amazon EBS root device volume of the Linux AMI that is used for each Amazon EC2 instance. Available in Amazon EMR releases 6.15.0 and later. |
ebs_root_volume_size | integer | The size, in GiB, of the Amazon EBS root device volume of the Linux AMI that is used for each Amazon EC2 instance. Available in Amazon EMR releases 4.x and later. |
ebs_root_volume_throughput | integer | The throughput, in MiB/s, of the Amazon EBS root device volume of the Linux AMI that is used for each Amazon EC2 instance. Available in Amazon EMR releases 6.15.0 and later. |
ec_2_instance_attributes | object | Provides information about the Amazon EC2 instances in a cluster grouped by category. For example, key name, subnet ID, IAM instance profile, and so on. |
extended_support | boolean | Reserved. |
id | string | The unique identifier for the cluster. |
instance_collection_type | string | The instance fleet configuration is available only in Amazon EMR releases 4.8.0 and later, excluding 5.0.x versions. The instance group configuration of the cluster. A value of INSTANCE_GROUP indicates a uniform instance group configuration. A value of INSTANCE_FLEET indicates an instance fleets configuration. (INSTANCE_FLEET, INSTANCE_GROUP) |
kerberos_attributes | object | Attributes for Kerberos configuration when Kerberos authentication is enabled using a security configuration. For more information see Use Kerberos Authentication in the Amazon EMR Management Guide. |
log_encryption_kms_key_id | string | The KMS key used for encrypting log files. This attribute is only available with Amazon EMR 5.30.0 and later, excluding Amazon EMR 6.0.0. |
log_uri | string | The path to the Amazon S3 location where logs for this cluster are stored. |
master_public_dns_name | string | The DNS name of the master node. If the cluster is on a private subnet, this is the private DNS name. On a public subnet, this is the public DNS name. |
monitoring_configuration | object | Contains Cloudwatch log configuration metadata and settings. |
name | string | The name of the cluster. This parameter can't contain the characters <, >, $, |, or ` (backtick). |
normalized_instance_hours | integer | An approximation of the cost of the cluster, represented in m1.small/hours. This value is incremented one time for every hour an m1.small instance runs. Larger instances are weighted more, so an Amazon EC2 instance that is roughly four times more expensive would result in the normalized instance hours being incremented by four. This result is only an approximation and does not reflect the actual billing rate. |
os_release_label | string | The Amazon Linux release specified in a cluster launch RunJobFlow request. If no Amazon Linux release was specified, the default Amazon Linux release is shown in the response. |
outpost_arn | string | The Amazon Resource Name (ARN) of the Outpost where the cluster is launched. |
placement_groups | array | Placement group configured for an Amazon EMR cluster. |
release_label | string | The Amazon EMR release label, which determines the version of open-source application packages installed on the cluster. Release labels are in the form emr-x.x.x, where x.x.x is an Amazon EMR release version such as emr-5.14.0. For more information about Amazon EMR release versions and included application versions and features, see https://docs.aws.amazon.com/emr/latest/ReleaseGuide/. The release label applies only to Amazon EMR releases version 4.0 and later. Earlier versions use AmiVersion. |
repo_upgrade_on_boot | string | Applies only when CustomAmiID is used. Specifies the type of updates that the Amazon Linux AMI package repositories apply when an instance boots using the AMI. (SECURITY, NONE) |
requested_ami_version | string | The AMI version requested for this cluster. |
running_ami_version | string | The AMI version running on this cluster. |
scale_down_behavior | string | The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized. TERMINATE_AT_INSTANCE_HOUR indicates that Amazon EMR terminates nodes at the instance-hour boundary, regardless of when the request to terminate the instance was submitted. This option is only available with Amazon EMR 5.1.0 and later and is the default for clusters created using that version. TERMINATE_AT_TASK_COMPLETION indicates that Amazon EMR adds nodes to a deny list and drains tasks from nodes before terminating the Amazon EC2 instances, regardless of the instance-hour boundary. With either behavior, Amazon EMR removes the least active nodes first and blocks instance termination if it could lead to HDFS corruption. TERMINATE_AT_TASK_COMPLETION is available only in Amazon EMR releases 4.1.0 and later, and is the default for versions of Amazon EMR earlier than 5.1.0. (TERMINATE_AT_INSTANCE_HOUR, TERMINATE_AT_TASK_COMPLETION) |
security_configuration | string | The name of the security configuration applied to the cluster. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>) |
service_role | string | The IAM role that Amazon EMR assumes in order to access Amazon Web Services resources on your behalf. |
session_enabled | boolean | Indicates whether Spark Connect sessions are enabled on the cluster. |
status | object | The current status details about the cluster. |
step_concurrency_level | integer | Specifies the number of steps that can be executed concurrently. |
tags | array | A list of tags associated with a cluster. |
termination_protected | boolean | Indicates whether Amazon EMR will lock the cluster to prevent the Amazon EC2 instances from being terminated by an API call or user intervention, or in the event of a cluster error. |
unhealthy_node_replacement | boolean | Indicates whether Amazon EMR should gracefully replace Amazon EC2 core instances that have degraded within the cluster. |
visible_to_all_users | boolean | Indicates whether the cluster is visible to IAM principals in the Amazon Web Services account associated with the cluster. When true, IAM principals in the Amazon Web Services account can perform Amazon EMR cluster actions on the cluster that their IAM policies allow. When false, only the IAM principal that created the cluster and the Amazon Web Services account root user can perform Amazon EMR actions, regardless of IAM permissions policies attached to other IAM principals. The default value is true if a value is not provided when creating a cluster using the Amazon EMR API RunJobFlow command, the CLI create-cluster command, or the Amazon Web Services Management Console. |
| Name | Datatype | Description |
|---|---|---|
cluster_arn | string | The Amazon Resource Name of the cluster. |
id | string | The unique identifier for the cluster. |
name | string | The name of the cluster. |
normalized_instance_hours | integer | An approximation of the cost of the cluster, represented in m1.small/hours. This value is incremented one time for every hour an m1.small instance runs. Larger instances are weighted more, so an Amazon EC2 instance that is roughly four times more expensive would result in the normalized instance hours being incremented by four. This result is only an approximation and does not reflect the actual billing rate. |
outpost_arn | string | The Amazon Resource Name (ARN) of the Outpost where the cluster is launched. |
status | object | The details about the current status of the cluster. |
Methods
The following methods are available for this resource:
| Name | Accessible by | Required Params | Optional Params | Description |
|---|---|---|---|---|
describe_cluster | select | region | Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on. | |
list_clusters | select | region | Provides the status of all clusters visible to this Amazon Web Services account. Allows you to filter the list of clusters based on certain criteria; for example, filtering by cluster creation date and time or by status. This call returns a maximum of 50 clusters in unsorted order per call, but returns a marker to track the paging of the cluster list across multiple ListClusters calls. | |
modify_cluster | update | region, ClusterId | Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID. | |
start_session | exec | region, ClusterId | Creates and starts a new Spark Connect session on the specified cluster. The cluster must be in the RUNNING or WAITING state and have sessions enabled. This operation is supported in Amazon EMR Spark 8.0.0 and later. |
Parameters
Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.
| Name | Datatype | Description |
|---|---|---|
region | string | AWS region (default: us-east-1) |
SELECT examples
- describe_cluster
- list_clusters
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
SELECT
applications,
auto_scaling_role,
auto_terminate,
cluster_arn,
configurations,
custom_ami_id,
ebs_root_volume_iops,
ebs_root_volume_size,
ebs_root_volume_throughput,
ec_2_instance_attributes,
extended_support,
id,
instance_collection_type,
kerberos_attributes,
log_encryption_kms_key_id,
log_uri,
master_public_dns_name,
monitoring_configuration,
name,
normalized_instance_hours,
os_release_label,
outpost_arn,
placement_groups,
release_label,
repo_upgrade_on_boot,
requested_ami_version,
running_ami_version,
scale_down_behavior,
security_configuration,
service_role,
session_enabled,
status,
step_concurrency_level,
tags,
termination_protected,
unhealthy_node_replacement,
visible_to_all_users
FROM aws.emr.clusters
WHERE region = '{{ region }}' -- required
;
Provides the status of all clusters visible to this Amazon Web Services account. Allows you to filter the list of clusters based on certain criteria; for example, filtering by cluster creation date and time or by status. This call returns a maximum of 50 clusters in unsorted order per call, but returns a marker to track the paging of the cluster list across multiple ListClusters calls.
SELECT
cluster_arn,
id,
name,
normalized_instance_hours,
outpost_arn,
status
FROM aws.emr.clusters
WHERE region = '{{ region }}' -- required
;
UPDATE examples
- modify_cluster
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
UPDATE aws.emr.clusters
SET
ClusterId = '{{ ClusterId }}',
StepConcurrencyLevel = {{ StepConcurrencyLevel }},
ExtendedSupport = {{ ExtendedSupport }}
WHERE
region = '{{ region }}' --required
AND ClusterId = '{{ ClusterId }}' --required
RETURNING
extended_support,
step_concurrency_level;
Lifecycle Methods
- start_session
Creates and starts a new Spark Connect session on the specified cluster. The cluster must be in the RUNNING or WAITING state and have sessions enabled. This operation is supported in Amazon EMR Spark 8.0.0 and later.
EXEC aws.emr.clusters.start_session
@region='{{ region }}' --required
@@json=
'{
"Name": "{{ Name }}",
"ClusterId": "{{ ClusterId }}",
"ExecutionRoleArn": "{{ ExecutionRoleArn }}",
"EngineConfigurations": "{{ EngineConfigurations }}",
"MonitoringConfiguration": "{{ MonitoringConfiguration }}",
"SessionIdleTimeoutInMinutes": {{ SessionIdleTimeoutInMinutes }},
"ClientRequestToken": "{{ ClientRequestToken }}",
"Tags": "{{ Tags }}"
}'
;