clusters

Creates, updates, deletes, gets or lists a clusters resource.

Overview

Name	clusters
Type	Resource
Id	aws.kafka.clusters

Fields

The following fields are returned by SELECT queries:

describe_cluster

Name	Datatype	Description
active_operation_arn	string	Arn of active cluster operation.
broker_node_group_info	object	Describes the setup to be used for Apache Kafka broker nodes in the cluster.
client_authentication	object	Includes all client authentication information.
cluster_arn	string	The Amazon Resource Name (ARN) that uniquely identifies the cluster.
cluster_name	string	The name of the cluster.
creation_time	string (date-time)	The time when the cluster was created.
current_broker_software_info	object	Information about the version of software currently deployed on the Apache Kafka brokers in the cluster.
current_version	string	The current version of the MSK cluster.
customer_action_status	string	Determines if there is an action required from the customer. (CRITICAL_ACTION_REQUIRED, ACTION_RECOMMENDED, NONE)
encryption_info	object	Includes encryption-related information, such as the AWS KMS key used for encrypting data at rest and whether you want MSK to encrypt your data in transit.
enhanced_monitoring	string	Specifies which metrics are gathered for the MSK cluster. This property has the following possible values: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, and PER_TOPIC_PER_PARTITION. For a list of the metrics associated with each of these levels of monitoring, see Monitoring. (DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, PER_TOPIC_PER_PARTITION)
logging_info	object
number_of_broker_nodes	integer	The number of broker nodes in the cluster.
open_monitoring	object	Settings for open monitoring using Prometheus.
rebalancing	object	Specifies whether or not intelligent rebalancing is turned on for a newly created MSK Provisioned cluster with Express brokers. Intelligent rebalancing performs automatic partition balancing operations when you scale your clusters up or down. By default, intelligent rebalancing is ACTIVE for all new Express-based clusters.
state	string	The state of the cluster. The possible states are ACTIVE, CREATING, DELETING, FAILED, HEALING, MAINTENANCE, REBOOTING_BROKER, and UPDATING. (ACTIVE, CREATING, DELETING, FAILED, HEALING, MAINTENANCE, REBOOTING_BROKER, UPDATING)
state_info	object
storage_mode	string	Controls storage mode for various supported storage tiers. (LOCAL, TIERED)
tags	object	Tags attached to the cluster.
zookeeper_connect_string	string	The connection string to use to connect to the Apache ZooKeeper cluster.
zookeeper_connect_string_tls	string	The connection string to use to connect to zookeeper cluster on Tls port.

list_clusters

Name	Datatype	Description
active_operation_arn	string	Arn of active cluster operation.
broker_node_group_info	object	Describes the setup to be used for Apache Kafka broker nodes in the cluster.
client_authentication	object	Includes all client authentication information.
cluster_arn	string	The Amazon Resource Name (ARN) that uniquely identifies the cluster.
cluster_name	string	The name of the cluster.
creation_time	string (date-time)	The time when the cluster was created.
current_broker_software_info	object	Information about the version of software currently deployed on the Apache Kafka brokers in the cluster.
current_version	string	The current version of the MSK cluster.
customer_action_status	string	Determines if there is an action required from the customer. (CRITICAL_ACTION_REQUIRED, ACTION_RECOMMENDED, NONE)
encryption_info	object	Includes encryption-related information, such as the AWS KMS key used for encrypting data at rest and whether you want MSK to encrypt your data in transit.
enhanced_monitoring	string	Specifies which metrics are gathered for the MSK cluster. This property has the following possible values: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, and PER_TOPIC_PER_PARTITION. For a list of the metrics associated with each of these levels of monitoring, see Monitoring. (DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, PER_TOPIC_PER_PARTITION)
logging_info	object
number_of_broker_nodes	integer	The number of broker nodes in the cluster.
open_monitoring	object	Settings for open monitoring using Prometheus.
rebalancing	object	Specifies whether or not intelligent rebalancing is turned on for a newly created MSK Provisioned cluster with Express brokers. Intelligent rebalancing performs automatic partition balancing operations when you scale your clusters up or down. By default, intelligent rebalancing is ACTIVE for all new Express-based clusters.
state	string	The state of the cluster. The possible states are ACTIVE, CREATING, DELETING, FAILED, HEALING, MAINTENANCE, REBOOTING_BROKER, and UPDATING. (ACTIVE, CREATING, DELETING, FAILED, HEALING, MAINTENANCE, REBOOTING_BROKER, UPDATING)
state_info	object
storage_mode	string	Controls storage mode for various supported storage tiers. (LOCAL, TIERED)
tags	object	Tags attached to the cluster.
zookeeper_connect_string	string	The connection string to use to connect to the Apache ZooKeeper cluster.
zookeeper_connect_string_tls	string	The connection string to use to connect to zookeeper cluster on Tls port.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
describe_cluster	select	cluster_arn, region		Returns a description of the MSK cluster whose Amazon Resource Name (ARN) is specified in the request.
list_clusters	select	region	clusterNameFilter, maxResults, nextToken	Returns a list of all the MSK clusters in the current Region.
create_cluster	insert	region, BrokerNodeGroupInfo, ClusterName, KafkaVersion, NumberOfBrokerNodes		Creates a new MSK cluster.
update_broker_count	update	cluster_arn, region, CurrentVersion, TargetNumberOfBrokerNodes		Updates the number of broker nodes in the cluster.
update_broker_storage	update	cluster_arn, region, CurrentVersion, TargetBrokerEBSVolumeInfo		Updates the EBS storage associated with MSK brokers.
update_broker_type	update	cluster_arn, region, CurrentVersion, TargetInstanceType		Updates EC2 instance type.
update_connectivity	update	cluster_arn, region, CurrentVersion		Updates the cluster's connectivity configuration.
delete_cluster	delete	cluster_arn, region	currentVersion	Deletes the MSK cluster specified by the Amazon Resource Name (ARN) in the request.
batch_associate_scram_secret	exec	cluster_arn, region, SecretArnList		Associates one or more Scram Secrets with an Amazon MSK cluster.
batch_disassociate_scram_secret	exec	cluster_arn, region, SecretArnList		Disassociates one or more Scram Secrets from an Amazon MSK cluster.
reboot_broker	exec	cluster_arn, region, BrokerIds		Reboots brokers.
update_monitoring	exec	cluster_arn, region, CurrentVersion		Updates the monitoring settings for the cluster. You can use this operation to specify which Apache Kafka metrics you want Amazon MSK to send to Amazon CloudWatch. You can also specify settings for open monitoring with Prometheus.
update_rebalancing	exec	cluster_arn, region, CurrentVersion		Use this resource to update the intelligent rebalancing status of an Amazon MSK Provisioned cluster with Express brokers.
update_security	exec	cluster_arn, region, CurrentVersion		Updates the security settings for the cluster. You can use this operation to specify encryption and authentication on existing clusters.
update_storage	exec	cluster_arn, region, CurrentVersion		Updates cluster broker volume size (or) sets cluster storage mode to TIERED.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
cluster_arn	string	The Amazon Resource Name (ARN) of the cluster to be updated.
region	string	AWS region (default: us-east-1)
clusterNameFilter	string	Specify a prefix of the name of the clusters that you want to list. The service lists all the clusters whose names start with this prefix.
currentVersion	string	The current version of the MSK cluster.
maxResults	integer	The maximum number of results to return in the response. If there are more results, the response includes a NextToken parameter.
nextToken	string	The paginated results marker. When the result of the operation is truncated, the call returns NextToken in the response. To get the next batch, provide this token in your next request.

SELECT examples

describe_cluster

Returns a description of the MSK cluster whose Amazon Resource Name (ARN) is specified in the request.

SELECT
active_operation_arn,
broker_node_group_info,
client_authentication,
cluster_arn,
cluster_name,
creation_time,
current_broker_software_info,
current_version,
customer_action_status,
encryption_info,
enhanced_monitoring,
logging_info,
number_of_broker_nodes,
open_monitoring,
rebalancing,
state,
state_info,
storage_mode,
tags,
zookeeper_connect_string,
zookeeper_connect_string_tls
FROM aws.kafka.clusters
WHERE cluster_arn = '{{ cluster_arn }}' -- required
AND region = '{{ region }}' -- required
;

list_clusters

Returns a list of all the MSK clusters in the current Region.

SELECT
active_operation_arn,
broker_node_group_info,
client_authentication,
cluster_arn,
cluster_name,
creation_time,
current_broker_software_info,
current_version,
customer_action_status,
encryption_info,
enhanced_monitoring,
logging_info,
number_of_broker_nodes,
open_monitoring,
rebalancing,
state,
state_info,
storage_mode,
tags,
zookeeper_connect_string,
zookeeper_connect_string_tls
FROM aws.kafka.clusters
WHERE region = '{{ region }}' -- required
AND clusterNameFilter = '{{ clusterNameFilter }}'
AND maxResults = '{{ maxResults }}'
AND nextToken = '{{ nextToken }}'
;

INSERT examples

create_cluster

Creates a new MSK cluster.

INSERT INTO aws.kafka.clusters (
BrokerNodeGroupInfo,
Rebalancing,
ClientAuthentication,
ClusterName,
ConfigurationInfo,
EncryptionInfo,
EnhancedMonitoring,
OpenMonitoring,
KafkaVersion,
LoggingInfo,
NumberOfBrokerNodes,
Tags,
StorageMode,
region
)
SELECT 
'{{ BrokerNodeGroupInfo }}' /* required */,
'{{ Rebalancing }}',
'{{ ClientAuthentication }}',
'{{ ClusterName }}' /* required */,
'{{ ConfigurationInfo }}',
'{{ EncryptionInfo }}',
'{{ EnhancedMonitoring }}',
'{{ OpenMonitoring }}',
'{{ KafkaVersion }}' /* required */,
'{{ LoggingInfo }}',
{{ NumberOfBrokerNodes }} /* required */,
'{{ Tags }}',
'{{ StorageMode }}',
'{{ region }}'
RETURNING
cluster_arn,
cluster_name,
state
;

Manifest

# Description fields are for documentation purposes
- name: clusters
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the clusters resource.
  - name: BrokerNodeGroupInfo
    description: |
      Describes the setup to be used for Apache Kafka broker nodes in the cluster.
    value:
      BrokerAZDistribution: "{{ BrokerAZDistribution }}"
      ClientSubnets:
        - "{{ ClientSubnets }}"
      InstanceType: "{{ InstanceType }}"
      SecurityGroups:
        - "{{ SecurityGroups }}"
      StorageInfo:
        EbsStorageInfo:
          ProvisionedThroughput:
            Enabled: {{ Enabled }}
            VolumeThroughput: {{ VolumeThroughput }}
          VolumeSize: {{ VolumeSize }}
      ConnectivityInfo:
        PublicAccess:
          Type: "{{ Type }}"
        VpcConnectivity:
          ClientAuthentication:
            Sasl:
              Scram: "{{ Scram }}"
              Iam: "{{ Iam }}"
            Tls:
              Enabled: {{ Enabled }}
        NetworkType: "{{ NetworkType }}"
      ZoneIds:
        - "{{ ZoneIds }}"
  - name: Rebalancing
    description: |
      Specifies whether or not intelligent rebalancing is turned on for a newly created MSK Provisioned cluster with Express brokers. Intelligent rebalancing performs automatic partition balancing operations when you scale your clusters up or down. By default, intelligent rebalancing is ACTIVE for all new Express-based clusters.
    value:
      Status: "{{ Status }}"
  - name: ClientAuthentication
    description: |
      Includes all client authentication information.
    value:
      Sasl:
        Scram:
          Enabled: {{ Enabled }}
        Iam:
          Enabled: {{ Enabled }}
      Tls:
        CertificateAuthorityArnList:
          - "{{ CertificateAuthorityArnList }}"
        Enabled: {{ Enabled }}
      Unauthenticated:
        Enabled: {{ Enabled }}
  - name: ClusterName
    value: "{{ ClusterName }}"
  - name: ConfigurationInfo
    description: |
      Specifies the configuration to use for the brokers.
    value:
      Arn: "{{ Arn }}"
      Revision: {{ Revision }}
  - name: EncryptionInfo
    description: |
      Includes encryption-related information, such as the AWS KMS key used for encrypting data at rest and whether you want MSK to encrypt your data in transit.
    value:
      EncryptionAtRest:
        DataVolumeKMSKeyId: "{{ DataVolumeKMSKeyId }}"
      EncryptionInTransit:
        ClientBroker: "{{ ClientBroker }}"
        InCluster: {{ InCluster }}
  - name: EnhancedMonitoring
    value: "{{ EnhancedMonitoring }}"
    description: |
      Specifies which metrics are gathered for the MSK cluster. This property has the following possible values: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER, and PER_TOPIC_PER_PARTITION. For a list of the metrics associated with each of these levels of monitoring, see Monitoring.
    valid_values: ['DEFAULT', 'PER_BROKER', 'PER_TOPIC_PER_BROKER', 'PER_TOPIC_PER_PARTITION']
  - name: OpenMonitoring
    description: |
      JMX and Node monitoring for the MSK cluster.
    value:
      Prometheus:
        JmxExporter:
          EnabledInBroker: {{ EnabledInBroker }}
        NodeExporter:
          EnabledInBroker: {{ EnabledInBroker }}
  - name: KafkaVersion
    value: "{{ KafkaVersion }}"
  - name: LoggingInfo
    value:
      AuthorizerLogs:
        CloudWatchLogs:
          Enabled: {{ Enabled }}
          LogGroup: "{{ LogGroup }}"
        Firehose:
          DeliveryStream: "{{ DeliveryStream }}"
          Enabled: {{ Enabled }}
        S3:
          Bucket: "{{ Bucket }}"
          Enabled: {{ Enabled }}
          Prefix: "{{ Prefix }}"
      BrokerLogs:
        CloudWatchLogs:
          Enabled: {{ Enabled }}
          LogGroup: "{{ LogGroup }}"
        Firehose:
          DeliveryStream: "{{ DeliveryStream }}"
          Enabled: {{ Enabled }}
        S3:
          Bucket: "{{ Bucket }}"
          Enabled: {{ Enabled }}
          Prefix: "{{ Prefix }}"
  - name: NumberOfBrokerNodes
    value: {{ NumberOfBrokerNodes }}
  - name: Tags
    value: "{{ Tags }}"
  - name: StorageMode
    value: "{{ StorageMode }}"
    description: |
      Controls storage mode for various supported storage tiers.
    valid_values: ['LOCAL', 'TIERED']

UPDATE examples

update_broker_count

Updates the number of broker nodes in the cluster.

UPDATE aws.kafka.clusters
SET 
CurrentVersion = '{{ CurrentVersion }}',
TargetNumberOfBrokerNodes = {{ TargetNumberOfBrokerNodes }}
WHERE 
cluster_arn = '{{ cluster_arn }}' --required
AND region = '{{ region }}' --required
AND CurrentVersion = '{{ CurrentVersion }}' --required
AND TargetNumberOfBrokerNodes = '{{ TargetNumberOfBrokerNodes }}' --required
RETURNING
cluster_arn,
cluster_operation_arn;

update_broker_storage

Updates the EBS storage associated with MSK brokers.

UPDATE aws.kafka.clusters
SET 
CurrentVersion = '{{ CurrentVersion }}',
TargetBrokerEBSVolumeInfo = '{{ TargetBrokerEBSVolumeInfo }}'
WHERE 
cluster_arn = '{{ cluster_arn }}' --required
AND region = '{{ region }}' --required
AND CurrentVersion = '{{ CurrentVersion }}' --required
AND TargetBrokerEBSVolumeInfo = '{{ TargetBrokerEBSVolumeInfo }}' --required
RETURNING
cluster_arn,
cluster_operation_arn;

update_broker_type

Updates EC2 instance type.

UPDATE aws.kafka.clusters
SET 
CurrentVersion = '{{ CurrentVersion }}',
TargetInstanceType = '{{ TargetInstanceType }}'
WHERE 
cluster_arn = '{{ cluster_arn }}' --required
AND region = '{{ region }}' --required
AND CurrentVersion = '{{ CurrentVersion }}' --required
AND TargetInstanceType = '{{ TargetInstanceType }}' --required
RETURNING
cluster_arn,
cluster_operation_arn;

update_connectivity

Updates the cluster's connectivity configuration.

UPDATE aws.kafka.clusters
SET 
ConnectivityInfo = '{{ ConnectivityInfo }}',
CurrentVersion = '{{ CurrentVersion }}',
ZookeeperAccess = '{{ ZookeeperAccess }}'
WHERE 
cluster_arn = '{{ cluster_arn }}' --required
AND region = '{{ region }}' --required
AND CurrentVersion = '{{ CurrentVersion }}' --required
RETURNING
cluster_arn,
cluster_operation_arn;

DELETE examples

delete_cluster

Deletes the MSK cluster specified by the Amazon Resource Name (ARN) in the request.

DELETE FROM aws.kafka.clusters
WHERE cluster_arn = '{{ cluster_arn }}' --required
AND region = '{{ region }}' --required
AND currentVersion = '{{ currentVersion }}'
;

Lifecycle Methods

batch_associate_scram_secret

Associates one or more Scram Secrets with an Amazon MSK cluster.

EXEC aws.kafka.clusters.batch_associate_scram_secret 
@cluster_arn='{{ cluster_arn }}' --required, 
@region='{{ region }}' --required 
@@json=
'{
"SecretArnList": "{{ SecretArnList }}"
}'
;

batch_disassociate_scram_secret

Disassociates one or more Scram Secrets from an Amazon MSK cluster.

EXEC aws.kafka.clusters.batch_disassociate_scram_secret 
@cluster_arn='{{ cluster_arn }}' --required, 
@region='{{ region }}' --required 
@@json=
'{
"SecretArnList": "{{ SecretArnList }}"
}'
;

reboot_broker

Reboots brokers.

EXEC aws.kafka.clusters.reboot_broker 
@cluster_arn='{{ cluster_arn }}' --required, 
@region='{{ region }}' --required 
@@json=
'{
"BrokerIds": "{{ BrokerIds }}"
}'
;

update_monitoring

Updates the monitoring settings for the cluster. You can use this operation to specify which Apache Kafka metrics you want Amazon MSK to send to Amazon CloudWatch. You can also specify settings for open monitoring with Prometheus.

EXEC aws.kafka.clusters.update_monitoring 
@cluster_arn='{{ cluster_arn }}' --required, 
@region='{{ region }}' --required 
@@json=
'{
"CurrentVersion": "{{ CurrentVersion }}", 
"EnhancedMonitoring": "{{ EnhancedMonitoring }}", 
"OpenMonitoring": "{{ OpenMonitoring }}", 
"LoggingInfo": "{{ LoggingInfo }}"
}'
;

update_rebalancing

Use this resource to update the intelligent rebalancing status of an Amazon MSK Provisioned cluster with Express brokers.

EXEC aws.kafka.clusters.update_rebalancing 
@cluster_arn='{{ cluster_arn }}' --required, 
@region='{{ region }}' --required 
@@json=
'{
"CurrentVersion": "{{ CurrentVersion }}", 
"Rebalancing": "{{ Rebalancing }}"
}'
;

update_security

Updates the security settings for the cluster. You can use this operation to specify encryption and authentication on existing clusters.

EXEC aws.kafka.clusters.update_security 
@cluster_arn='{{ cluster_arn }}' --required, 
@region='{{ region }}' --required 
@@json=
'{
"ClientAuthentication": "{{ ClientAuthentication }}", 
"CurrentVersion": "{{ CurrentVersion }}", 
"EncryptionInfo": "{{ EncryptionInfo }}"
}'
;

update_storage

Updates cluster broker volume size (or) sets cluster storage mode to TIERED.

EXEC aws.kafka.clusters.update_storage 
@cluster_arn='{{ cluster_arn }}' --required, 
@region='{{ region }}' --required 
@@json=
'{
"CurrentVersion": "{{ CurrentVersion }}", 
"ProvisionedThroughput": "{{ ProvisionedThroughput }}", 
"StorageMode": "{{ StorageMode }}", 
"VolumeSizeGB": {{ VolumeSizeGB }}
}'
;