Skip to main content

cluster_nodes

Creates, updates, deletes, gets or lists a cluster_nodes resource.

Overview​

Namecluster_nodes
TypeResource
Idaws.sagemaker.cluster_nodes

Fields​

The following fields are returned by SELECT queries:

NameDatatypeDescription
capacity_typestringThe capacity type of the node. Valid values are OnDemand and Spot. When set to OnDemand, the node is launched as an On-Demand instance. When set to Spot, the node is launched as a Spot instance. (Spot, OnDemand)
current_image_idstringThe ID of the Amazon Machine Image (AMI) currently in use by the node. (pattern: <code>ami-[0-9a-fA-F]{8,17}|default</code>)
current_image_release_versionstringThe version of the HyperPod-managed AMI currently running on the node. (pattern: <code>[0-9]+.[0-9]+.[0-9]+</code>)
desired_image_idstringThe ID of the Amazon Machine Image (AMI) desired for the node. (pattern: <code>ami-[0-9a-fA-F]{8,17}|default</code>)
desired_image_release_versionstringThe desired version of the HyperPod-managed AMI for the node. This may differ from the current version when an update is pending. (pattern: <code>[0-9]+.[0-9]+.[0-9]+</code>)
image_version_statusstringThe status of the image version for the cluster node. (UpToDate, UpdateAvailable, SecurityUpdateRequired, EndOfLife)
instance_group_namestringThe instance group name in which the instance is. (pattern: <code>[a-zA-Z0-9](-[a-zA-Z0-9])</code>)
instance_idstringThe ID of the instance.
instance_statusobjectThe status of the instance.
instance_storage_configsarrayThe configurations of additional storage specified to the instance group where the instance (node) is launched.
instance_typestringThe type of the instance. (ml.p4d.24xlarge, ml.p4de.24xlarge, ml.p5.48xlarge, ml.p5.4xlarge, ml.p6e-gb200.36xlarge, ml.trn1.32xlarge, ml.trn1n.32xlarge, ml.g5.xlarge, ml.g5.2xlarge, ml.g5.4xlarge, ml.g5.8xlarge, ml.g5.12xlarge, ml.g5.16xlarge, ml.g5.24xlarge, ml.g5.48xlarge, ml.c5.large, ml.c5.xlarge, ml.c5.2xlarge, ml.c5.4xlarge, ml.c5.9xlarge, ml.c5.12xlarge, ml.c5.18xlarge, ml.c5.24xlarge, ml.c5n.large, ml.c5n.2xlarge, ml.c5n.4xlarge, ml.c5n.9xlarge, ml.c5n.18xlarge, ml.m5.large, ml.m5.xlarge, ml.m5.2xlarge, ml.m5.4xlarge, ml.m5.8xlarge, ml.m5.12xlarge, ml.m5.16xlarge, ml.m5.24xlarge, ml.t3.medium, ml.t3.large, ml.t3.xlarge, ml.t3.2xlarge, ml.g6.xlarge, ml.g6.2xlarge, ml.g6.4xlarge, ml.g6.8xlarge, ml.g6.16xlarge, ml.g6.12xlarge, ml.g6.24xlarge, ml.g6.48xlarge, ml.gr6.4xlarge, ml.gr6.8xlarge, ml.g6e.xlarge, ml.g6e.2xlarge, ml.g6e.4xlarge, ml.g6e.8xlarge, ml.g6e.16xlarge, ml.g6e.12xlarge, ml.g6e.24xlarge, ml.g6e.48xlarge, ml.p5e.48xlarge, ml.p5en.48xlarge, ml.p6-b200.48xlarge, ml.trn2.3xlarge, ml.trn2.48xlarge, ml.c6i.large, ml.c6i.xlarge, ml.c6i.2xlarge, ml.c6i.4xlarge, ml.c6i.8xlarge, ml.c6i.12xlarge, ml.c6i.16xlarge, ml.c6i.24xlarge, ml.c6i.32xlarge, ml.m6i.large, ml.m6i.xlarge, ml.m6i.2xlarge, ml.m6i.4xlarge, ml.m6i.8xlarge, ml.m6i.12xlarge, ml.m6i.16xlarge, ml.m6i.24xlarge, ml.m6i.32xlarge, ml.r6i.large, ml.r6i.xlarge, ml.r6i.2xlarge, ml.r6i.4xlarge, ml.r6i.8xlarge, ml.r6i.12xlarge, ml.r6i.16xlarge, ml.r6i.24xlarge, ml.r6i.32xlarge, ml.i3en.large, ml.i3en.xlarge, ml.i3en.2xlarge, ml.i3en.3xlarge, ml.i3en.6xlarge, ml.i3en.12xlarge, ml.i3en.24xlarge, ml.m7i.large, ml.m7i.xlarge, ml.m7i.2xlarge, ml.m7i.4xlarge, ml.m7i.8xlarge, ml.m7i.12xlarge, ml.m7i.16xlarge, ml.m7i.24xlarge, ml.m7i.48xlarge, ml.r7i.large, ml.r7i.xlarge, ml.r7i.2xlarge, ml.r7i.4xlarge, ml.r7i.8xlarge, ml.r7i.12xlarge, ml.r7i.16xlarge, ml.r7i.24xlarge, ml.r7i.48xlarge, ml.r5d.16xlarge, ml.g7e.2xlarge, ml.g7e.4xlarge, ml.g7e.8xlarge, ml.g7e.12xlarge, ml.g7e.24xlarge, ml.g7e.48xlarge, ml.p6-b300.48xlarge, ml.g4dn.xlarge, ml.g4dn.2xlarge, ml.g4dn.4xlarge, ml.g4dn.8xlarge, ml.g4dn.12xlarge, ml.g4dn.16xlarge, ml.c6g.medium, ml.c6g.large, ml.c6g.xlarge, ml.c6g.2xlarge, ml.c6g.4xlarge, ml.c6g.8xlarge, ml.c6g.12xlarge, ml.c6g.16xlarge, ml.c7g.medium, ml.c7g.large, ml.c7g.xlarge, ml.c7g.2xlarge, ml.c7g.4xlarge, ml.c7g.8xlarge, ml.c7g.12xlarge, ml.c7g.16xlarge, ml.c8g.medium, ml.c8g.large, ml.c8g.xlarge, ml.c8g.2xlarge, ml.c8g.4xlarge, ml.c8g.8xlarge, ml.c8g.12xlarge, ml.c8g.16xlarge, ml.c8g.24xlarge, ml.c8g.48xlarge, ml.c6a.large, ml.c6a.xlarge, ml.c6a.2xlarge, ml.c6a.4xlarge, ml.c6a.8xlarge, ml.c6a.12xlarge, ml.c6a.16xlarge, ml.c6a.24xlarge, ml.c6a.32xlarge, ml.c6a.48xlarge, ml.m6a.large, ml.m6a.xlarge, ml.m6a.2xlarge, ml.m6a.4xlarge, ml.m6a.8xlarge, ml.m6a.12xlarge, ml.m6a.16xlarge, ml.m6a.24xlarge, ml.m6a.32xlarge, ml.m6a.48xlarge, ml.m6g.medium, ml.m6g.large, ml.m6g.xlarge, ml.m6g.2xlarge, ml.m6g.4xlarge, ml.m6g.8xlarge, ml.m6g.12xlarge, ml.m6g.16xlarge, ml.m7g.medium, ml.m7g.large, ml.m7g.xlarge, ml.m7g.2xlarge, ml.m7g.4xlarge, ml.m7g.8xlarge, ml.m7g.12xlarge, ml.m7g.16xlarge, ml.m8g.medium, ml.m8g.large, ml.m8g.xlarge, ml.m8g.2xlarge, ml.m8g.4xlarge, ml.m8g.8xlarge, ml.m8g.12xlarge, ml.m8g.16xlarge, ml.m8g.24xlarge, ml.m8g.48xlarge, ml.g7.2xlarge, ml.g7.4xlarge, ml.g7.8xlarge, ml.g7.12xlarge, ml.g7.24xlarge, ml.g7.48xlarge)
kubernetes_configobjectThe Kubernetes configuration applied to this node, showing both the current and desired state of labels and taints. The cluster works to reconcile the actual state with the declared state.
last_software_update_timestring (date-time)The time when the cluster was last updated.
launch_timestring (date-time)The time when the instance is launched.
life_cycle_configobjectThe LifeCycle configuration applied to the instance.
network_interfaceobjectThe network interface configuration for the cluster node.
node_logical_idstringA unique identifier for the node that persists throughout its lifecycle, from provisioning request to termination. This identifier can be used to track the node even before it has an assigned InstanceId. (pattern: <code>[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9]</code>)
override_vpc_configobjectSpecifies an Amazon Virtual Private Cloud (VPC) that your SageMaker jobs, hosted models, and compute resources have access to. You can control access to and from your resources by configuring a VPC. For more information, see Give SageMaker Access to Resources in your Amazon VPC.
placementobjectThe placement details of the SageMaker HyperPod cluster node.
private_dns_hostnamestringThe private DNS hostname of the SageMaker HyperPod cluster node. (pattern: <code>ip-((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)-?\b){4}..*</code>)
private_primary_ipstringThe private primary IP address of the SageMaker HyperPod cluster node. (pattern: <code>((25[0-5]|(2[0-4]|1\d|[1-9]|)\d).?\b){4}</code>)
private_primary_ipv_6stringThe private primary IPv6 address of the SageMaker HyperPod cluster node when configured with an Amazon VPC that supports IPv6 and includes subnets with IPv6 addressing enabled in either the cluster Amazon VPC configuration or the instance group Amazon VPC configuration.
threads_per_coreintegerThe number of threads per CPU core you specified under CreateCluster.
ultra_server_infoobjectContains information about the UltraServer.

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
describe_cluster_nodeselectregionRetrieves information of a node (also called a instance interchangeably) of a SageMaker HyperPod cluster.
attach_cluster_node_volumeupdateregion, ClusterArn, NodeId, VolumeIdAttaches your Amazon Elastic Block Store (Amazon EBS) volume to a node in your EKS orchestrated HyperPod cluster. This API works with the Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver to manage the lifecycle of persistent storage in your HyperPod EKS clusters.
batch_add_cluster_nodesexecregion, ClusterName, NodesToAddAdds nodes to a HyperPod cluster by incrementing the target count for one or more instance groups. This operation returns a unique NodeLogicalId for each node being added, which can be used to track the provisioning status of the node. This API provides a safer alternative to UpdateCluster for scaling operations by avoiding unintended configuration changes. This API is only supported for clusters using Continuous as the NodeProvisioningMode.
batch_delete_cluster_nodesexecregion, ClusterNameDeletes specific nodes within a SageMaker HyperPod cluster. BatchDeleteClusterNodes accepts a cluster name and a list of node IDs. To safeguard your work, back up your data to Amazon S3 or an FSx for Lustre file system before invoking the API on a worker node group. This will help prevent any potential data loss from the instance root volume. For more information about backup, see Use the backup script provided by SageMaker HyperPod. If you want to invoke this API on an existing cluster, you'll first need to patch the cluster by running the UpdateClusterSoftware API. For more information about patching a cluster, see Update the SageMaker HyperPod platform software of a cluster.
batch_reboot_cluster_nodesexecregion, ClusterNameReboots specific nodes within a SageMaker HyperPod cluster using a soft recovery mechanism. BatchRebootClusterNodes performs a graceful reboot of the specified nodes by calling the Amazon Elastic Compute Cloud RebootInstances API, which attempts to cleanly shut down the operating system before restarting the instance. This operation is useful for recovering from transient issues or applying certain configuration changes that require a restart. Rebooting a node may cause temporary service interruption for workloads running on that node. Ensure your workloads can handle node restarts or use appropriate scheduling to minimize impact. You can reboot up to 25 nodes in a single request. For SageMaker HyperPod clusters using the Slurm workload manager, ensure rebooting nodes will not disrupt critical cluster operations.
batch_replace_cluster_nodesexecregion, ClusterNameReplaces specific nodes within a SageMaker HyperPod cluster with new hardware. BatchReplaceClusterNodes terminates the specified instances and provisions new replacement instances with the same configuration but fresh hardware. The Amazon Machine Image (AMI) and instance configuration remain the same. This operation is useful for recovering from hardware failures or persistent issues that cannot be resolved through a reboot. Data Loss Warning: Replacing nodes destroys all instance volumes, including both root and secondary volumes. All data stored on these volumes will be permanently lost and cannot be recovered. To safeguard your work, back up your data to Amazon S3 or an FSx for Lustre file system before invoking the API on a worker node group. This will help prevent any potential data loss from the instance root volume. For more information about backup, see Use the backup script provided by SageMaker HyperPod. If you want to invoke this API on an existing cluster, you'll first need to patch the cluster by running the UpdateClusterSoftware API. For more information about patching a cluster, see Update the SageMaker HyperPod platform software of a cluster. You can replace up to 25 nodes in a single request.
detach_cluster_node_volumeexecregion, ClusterArn, NodeId, VolumeIdDetaches your Amazon Elastic Block Store (Amazon EBS) volume from a node in your EKS orchestrated SageMaker HyperPod cluster. This API works with the Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver to manage the lifecycle of persistent storage in your HyperPod EKS clusters.
list_cluster_nodesexecregion, ClusterNameRetrieves the list of instances (also called nodes interchangeably) in a SageMaker HyperPod cluster.

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
regionstringAWS region (default: us-east-1)

SELECT examples​

Retrieves information of a node (also called a instance interchangeably) of a SageMaker HyperPod cluster.

SELECT
capacity_type,
current_image_id,
current_image_release_version,
desired_image_id,
desired_image_release_version,
image_version_status,
instance_group_name,
instance_id,
instance_status,
instance_storage_configs,
instance_type,
kubernetes_config,
last_software_update_time,
launch_time,
life_cycle_config,
network_interface,
node_logical_id,
override_vpc_config,
placement,
private_dns_hostname,
private_primary_ip,
private_primary_ipv_6,
threads_per_core,
ultra_server_info
FROM aws.sagemaker.cluster_nodes
WHERE region = '{{ region }}' -- required
;

UPDATE examples​

Attaches your Amazon Elastic Block Store (Amazon EBS) volume to a node in your EKS orchestrated HyperPod cluster. This API works with the Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver to manage the lifecycle of persistent storage in your HyperPod EKS clusters.

UPDATE aws.sagemaker.cluster_nodes
SET 
ClusterArn = '{{ ClusterArn }}',
NodeId = '{{ NodeId }}',
VolumeId = '{{ VolumeId }}'
WHERE 
region = '{{ region }}' --required
AND ClusterArn = '{{ ClusterArn }}' --required
AND NodeId = '{{ NodeId }}' --required
AND VolumeId = '{{ VolumeId }}' --required
RETURNING
attach_time,
cluster_arn,
device_name,
node_id,
status,
volume_id;

Lifecycle Methods​

Adds nodes to a HyperPod cluster by incrementing the target count for one or more instance groups. This operation returns a unique NodeLogicalId for each node being added, which can be used to track the provisioning status of the node. This API provides a safer alternative to UpdateCluster for scaling operations by avoiding unintended configuration changes. This API is only supported for clusters using Continuous as the NodeProvisioningMode.

EXEC aws.sagemaker.cluster_nodes.batch_add_cluster_nodes 
@region='{{ region }}' --required 
@@json=
'{
"ClusterName": "{{ ClusterName }}", 
"ClientToken": "{{ ClientToken }}", 
"NodesToAdd": "{{ NodesToAdd }}"
}'
;
Last updated on