Skip to main content

training_jobs

Creates, updates, deletes, gets or lists a training_jobs resource.

Overview​

Nametraining_jobs
TypeResource
Idaws.sagemaker.training_jobs

Fields​

The following fields are returned by SELECT queries:

NameDatatypeDescription
algorithm_specificationobjectInformation about the algorithm used for training, and algorithm metadata.
auto_ml_job_arnstringThe Amazon Resource Name (ARN) of an AutoML job. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:automl-job/.*</code>)
billable_time_in_secondsintegerThe billable time in seconds. Billable time refers to the absolute wall-clock time. Multiply BillableTimeInSeconds by the number of instances (InstanceCount) in your training cluster to get the total compute time SageMaker bills you if you run distributed training. The formula is as follows: BillableTimeInSeconds * InstanceCount . You can calculate the savings from using managed spot training using the formula (1 - BillableTimeInSeconds / TrainingTimeInSeconds) * 100. For example, if BillableTimeInSeconds is 100 and TrainingTimeInSeconds is 500, the savings is 80%.
billable_token_countinteger (int64)The billable token count for eligible serverless training jobs.
checkpoint_configobjectContains information about the output location for managed spot training checkpoint data.
creation_timestring (date-time)A timestamp that indicates when the training job was created.
debug_hook_configobjectConfiguration information for the Amazon SageMaker Debugger hook parameters, metric and tensor collections, and storage paths. To learn more about how to configure the DebugHookConfig parameter, see Use the SageMaker and Debugger Configuration API Operations to Create, Update, and Debug Your Training Job.
debug_rule_configurationsarrayConfiguration information for Amazon SageMaker Debugger rules for debugging output tensors.
debug_rule_evaluation_statusesarrayEvaluation status of Amazon SageMaker Debugger rules for debugging on a training job.
enable_inter_container_traffic_encryptionbooleanTo encrypt all communications between ML compute instances in distributed training, choose True. Encryption provides greater security for distributed training, but training might take longer. How long it takes depends on the amount of communication between compute instances, especially if you use a deep learning algorithms in distributed training.
enable_managed_spot_trainingbooleanA Boolean indicating whether managed spot training is enabled (True) or not (False).
enable_network_isolationbooleanIf you want to allow inbound or outbound network calls, except for calls between peers within a training cluster for distributed training, choose True. If you enable network isolation for training jobs that are configured to use a VPC, SageMaker downloads and uploads customer data and model artifacts through the specified VPC, but the training container does not have network access.
environmentobjectThe environment variables to set in the Docker container. Do not include any security-sensitive information including account access IDs, secrets, or tokens in any environment fields. As part of the shared responsibility model, you are responsible for any potential exposure, unauthorized access, or compromise of your sensitive data if caused by security-sensitive information included in the request environment variable or plain text fields.
experiment_configobjectAssociates a SageMaker job as a trial component with an experiment and trial. Specified when you call the following APIs: CreateProcessingJob CreateTrainingJob CreateTransformJob
failure_reasonstringIf the training job failed, the reason it failed.
final_metric_data_listarrayA collection of MetricData objects that specify the names, values, and dates and times that the training algorithm emitted to Amazon CloudWatch.
hyper_parametersobjectAlgorithm-specific parameters.
infra_check_configobjectContains information about the infrastructure health check configuration for the training job.
input_data_configarrayAn array of Channel objects that describes each data input channel.
labeling_job_arnstringThe Amazon Resource Name (ARN) of the SageMaker Ground Truth labeling job that created the transform or training job. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:labeling-job/.*</code>)
last_modified_timestring (date-time)A timestamp that indicates when the status of the training job was last modified.
mlflow_configobjectThe MLflow configuration using SageMaker managed MLflow.
mlflow_detailsobjectThe MLflow details of this job.
model_artifactsobjectInformation about the Amazon S3 location that is configured for storing model artifacts.
model_package_configobjectThe configuration for the model package.
output_data_configobjectThe S3 path where model artifacts that you configured when creating the job are stored. SageMaker creates subfolders for model artifacts.
output_model_package_arnstringThe Amazon Resource Name (ARN) of the output model package containing model weights or checkpoints. (pattern: <code>arn:aws[a-z-]*:sagemaker:[a-z0-9-]{9,16}:[0-9]{12}:model-package/[\S]{1,2048}</code>)
profiler_configobjectConfiguration information for Amazon SageMaker Debugger system monitoring, framework profiling, and storage paths.
profiler_rule_configurationsarrayConfiguration information for Amazon SageMaker Debugger rules for profiling system and framework metrics.
profiler_rule_evaluation_statusesarrayEvaluation status of Amazon SageMaker Debugger rules for profiling on a training job.
profiling_statusstringProfiling status of a training job. (Enabled, Disabled)
progress_infoobjectThe Serverless training job progress information.
remote_debug_configobjectConfiguration for remote debugging. To learn more about the remote debugging functionality of SageMaker, see Access a training container through Amazon Web Services Systems Manager (SSM) for remote debugging.
resource_configobjectResources, including ML compute instances and ML storage volumes, that are configured for model training.
retry_strategyobjectThe number of times to retry the job when the job fails due to an InternalServerError.
role_arnstringThe Amazon Web Services Identity and Access Management (IAM) role configured for the training job. (pattern: <code>arn:aws[a-z-]*:iam::\d{12}:role/?[a-zA-Z_0-9+=,.@-_/]+</code>)
secondary_statusstringProvides detailed information about the state of the training job. For detailed information on the secondary status of the training job, see StatusMessage under SecondaryStatusTransition. SageMaker provides primary statuses and secondary statuses that apply to each of them: InProgress Starting - Starting the training job. Pending - The training job is waiting for compute capacity or compute resource provision. Downloading - An optional stage for algorithms that support File training input mode. It indicates that data is being downloaded to the ML storage volumes. Training - Training is in progress. Interrupted - The job stopped because the managed spot training instances were interrupted. Uploading - Training is complete and the model artifacts are being uploaded to the S3 location. Completed Completed - The training job has completed. Failed Failed - The training job has failed. The reason for the failure is returned in the FailureReason field of DescribeTrainingJobResponse. Stopped MaxRuntimeExceeded - The job stopped because it exceeded the maximum allowed runtime. MaxWaitTimeExceeded - The job stopped because it exceeded the maximum allowed wait time. Stopped - The training job has stopped. Stopping Stopping - Stopping the training job. Valid values for SecondaryStatus are subject to change. We no longer support the following secondary statuses: LaunchingMLInstances PreparingTraining DownloadingTrainingImage (Starting, LaunchingMLInstances, PreparingTrainingStack, Downloading, DownloadingTrainingImage, Training, Uploading, Stopping, Stopped, MaxRuntimeExceeded, Completed, Failed, Interrupted, MaxWaitTimeExceeded, Updating, Restarting, Pending)
secondary_status_transitionsarrayA history of all of the secondary statuses that the training job has transitioned through.
serverless_job_configobjectThe configuration for serverless training jobs.
stopping_conditionobjectSpecifies a limit to how long a job can run. When the job reaches the time limit, SageMaker ends the job. Use this API to cap costs. To stop a training job, SageMaker sends the algorithm the SIGTERM signal, which delays job termination for 120 seconds. Algorithms can use this 120-second window to save the model artifacts, so the results of training are not lost. The training algorithms provided by SageMaker automatically save the intermediate results of a model training job when possible. This attempt to save artifacts is only a best effort case as model might not be in a state from which it can be saved. For example, if training has just started, the model might not be ready to save. When saved, this intermediate data is a valid model artifact. You can use it to create a model with CreateModel. The Neural Topic Model (NTM) currently does not support saving intermediate model artifacts. When training NTMs, make sure that the maximum runtime is sufficient for the training job to complete.
tensor_board_output_configobjectConfiguration of storage locations for the Amazon SageMaker Debugger TensorBoard output data.
training_end_timestring (date-time)Indicates the time when the training job ends on training instances. You are billed for the time interval between the value of TrainingStartTime and this time. For successful jobs and stopped jobs, this is the time after model artifacts are uploaded. For failed jobs, this is the time when SageMaker detects a job failure.
training_job_arnstringThe Amazon Resource Name (ARN) of the training job. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:training-job/[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}</code>)
training_job_namestringName of the model training job. (pattern: <code>[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}</code>)
training_job_statusstringThe status of the training job. SageMaker provides the following training job statuses: InProgress - The training is in progress. Completed - The training job has completed. Failed - The training job has failed. To see the reason for the failure, see the FailureReason field in the response to a DescribeTrainingJobResponse call. Stopping - The training job is stopping. Stopped - The training job has stopped. For more detailed information, see SecondaryStatus. (InProgress, Completed, Failed, Stopping, Stopped, Deleting)
training_start_timestring (date-time)Indicates the time when the training job starts on training instances. You are billed for the time interval between this time and the value of TrainingEndTime. The start time in CloudWatch Logs might be later than this time. The difference is due to the time it takes to download the training data and to the size of the training container.
training_time_in_secondsintegerThe training time in seconds.
tuning_job_arnstringThe Amazon Resource Name (ARN) of the associated hyperparameter tuning job if the training job was launched by a hyperparameter tuning job. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:hyper-parameter-tuning-job/.*</code>)
vpc_configobjectSpecifies an Amazon Virtual Private Cloud (VPC) that your SageMaker jobs, hosted models, and compute resources have access to. You can control access to and from your resources by configuring a VPC. For more information, see Give SageMaker Access to Resources in your Amazon VPC.
warm_pool_statusobjectThe status of the warm pool associated with the training job.

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
describe_training_jobselectregionReturns information about a training job. Some of the attributes below only appear if the training job successfully starts. If the training job fails, TrainingJobStatus is Failed and, depending on the FailureReason, attributes like TrainingStartTime, TrainingTimeInSeconds, TrainingEndTime, and BillableTimeInSeconds may not be present in the response.
list_training_jobsselectregionLists training jobs. When StatusEquals and MaxResults are set at the same time, the MaxResults number of training jobs are first retrieved ignoring the StatusEquals parameter and then they are filtered by the StatusEquals parameter, which is returned as a response. For example, if ListTrainingJobs is invoked with the following parameters: { ... MaxResults: 100, StatusEquals: InProgress ... } First, 100 trainings jobs with any status, including those other than InProgress, are selected (sorted according to the creation time, from the most current to the oldest). Next, those with a status of InProgress are returned. You can quickly test the API using the following Amazon Web Services CLI code. aws sagemaker list-training-jobs --max-results 100 --status-equals InProgress
create_training_jobinsertregion, TrainingJobName, RoleArn, OutputDataConfigStarts a model training job. After training completes, SageMaker saves the resulting model artifacts to an Amazon S3 location that you specify. If you choose to host your model using SageMaker hosting services, you can use the resulting model artifacts as part of the model. You can also use the artifacts in a machine learning service other than SageMaker, provided that you know how to use them for inference. In the request body, you provide the following: AlgorithmSpecification - Identifies the training algorithm to use. HyperParameters - Specify these algorithm-specific parameters to enable the estimation of model parameters during training. Hyperparameters can be tuned to optimize this learning process. For a list of hyperparameters for each training algorithm provided by SageMaker, see Algorithms. Do not include any security-sensitive information including account access IDs, secrets, or tokens in any hyperparameter fields. As part of the shared responsibility model, you are responsible for any potential exposure, unauthorized access, or compromise of your sensitive data if caused by security-sensitive information included in the request hyperparameter variable or plain text fields. InputDataConfig - Describes the input required by the training job and the Amazon S3, EFS, or FSx location where it is stored. OutputDataConfig - Identifies the Amazon S3 bucket where you want SageMaker to save the results of model training. ResourceConfig - Identifies the resources, ML compute instances, and ML storage volumes to deploy for model training. In distributed training, you specify more than one instance. EnableManagedSpotTraining - Optimize the cost of training machine learning models by up to 80% by using Amazon EC2 Spot instances. For more information, see Managed Spot Training. RoleArn - The Amazon Resource Name (ARN) that SageMaker assumes to perform tasks on your behalf during model training. You must grant this role the necessary permissions so that SageMaker can successfully complete model training. StoppingCondition - To help cap training costs, use MaxRuntimeInSeconds to set a time limit for training. Use MaxWaitTimeInSeconds to specify how long a managed spot training job has to complete. Environment - The environment variables to set in the Docker container. Do not include any security-sensitive information including account access IDs, secrets, or tokens in any environment fields. As part of the shared responsibility model, you are responsible for any potential exposure, unauthorized access, or compromise of your sensitive data if caused by security-sensitive information included in the request environment variable or plain text fields. RetryStrategy - The number of times to retry the job when the job fails due to an InternalServerError. For more information about SageMaker, see How It Works.
update_training_jobupdateregion, TrainingJobNameUpdate a model training job to request a new Debugger profiling configuration or to change warm pool retention length.
delete_training_jobdeleteregionDeletes a training job. After SageMaker deletes a training job, all of the metadata for the training job is lost. You can delete only training jobs that are in a terminal state (Stopped, Failed, or Completed) and don't retain an Available managed warm pool. You cannot delete a job that is in the InProgress or Stopping state. After deleting the job, you can reuse its name to create another training job.
stop_training_jobexecregion, TrainingJobNameStops a training job. To stop a job, SageMaker sends the algorithm the SIGTERM signal, which delays job termination for 120 seconds. Algorithms might use this 120-second window to save the model artifacts, so the results of the training is not lost. When it receives a StopTrainingJob request, SageMaker changes the status of the job to Stopping. After SageMaker stops the job, it sets the status to Stopped.

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
regionstringAWS region (default: us-east-1)

SELECT examples​

Returns information about a training job. Some of the attributes below only appear if the training job successfully starts. If the training job fails, TrainingJobStatus is Failed and, depending on the FailureReason, attributes like TrainingStartTime, TrainingTimeInSeconds, TrainingEndTime, and BillableTimeInSeconds may not be present in the response.

SELECT
algorithm_specification,
auto_ml_job_arn,
billable_time_in_seconds,
billable_token_count,
checkpoint_config,
creation_time,
debug_hook_config,
debug_rule_configurations,
debug_rule_evaluation_statuses,
enable_inter_container_traffic_encryption,
enable_managed_spot_training,
enable_network_isolation,
environment,
experiment_config,
failure_reason,
final_metric_data_list,
hyper_parameters,
infra_check_config,
input_data_config,
labeling_job_arn,
last_modified_time,
mlflow_config,
mlflow_details,
model_artifacts,
model_package_config,
output_data_config,
output_model_package_arn,
profiler_config,
profiler_rule_configurations,
profiler_rule_evaluation_statuses,
profiling_status,
progress_info,
remote_debug_config,
resource_config,
retry_strategy,
role_arn,
secondary_status,
secondary_status_transitions,
serverless_job_config,
stopping_condition,
tensor_board_output_config,
training_end_time,
training_job_arn,
training_job_name,
training_job_status,
training_start_time,
training_time_in_seconds,
tuning_job_arn,
vpc_config,
warm_pool_status
FROM aws.sagemaker.training_jobs
WHERE region = '{{ region }}' -- required
;

INSERT examples​

Starts a model training job. After training completes, SageMaker saves the resulting model artifacts to an Amazon S3 location that you specify. If you choose to host your model using SageMaker hosting services, you can use the resulting model artifacts as part of the model. You can also use the artifacts in a machine learning service other than SageMaker, provided that you know how to use them for inference. In the request body, you provide the following: AlgorithmSpecification - Identifies the training algorithm to use. HyperParameters - Specify these algorithm-specific parameters to enable the estimation of model parameters during training. Hyperparameters can be tuned to optimize this learning process. For a list of hyperparameters for each training algorithm provided by SageMaker, see Algorithms. Do not include any security-sensitive information including account access IDs, secrets, or tokens in any hyperparameter fields. As part of the shared responsibility model, you are responsible for any potential exposure, unauthorized access, or compromise of your sensitive data if caused by security-sensitive information included in the request hyperparameter variable or plain text fields. InputDataConfig - Describes the input required by the training job and the Amazon S3, EFS, or FSx location where it is stored. OutputDataConfig - Identifies the Amazon S3 bucket where you want SageMaker to save the results of model training. ResourceConfig - Identifies the resources, ML compute instances, and ML storage volumes to deploy for model training. In distributed training, you specify more than one instance. EnableManagedSpotTraining - Optimize the cost of training machine learning models by up to 80% by using Amazon EC2 Spot instances. For more information, see Managed Spot Training. RoleArn - The Amazon Resource Name (ARN) that SageMaker assumes to perform tasks on your behalf during model training. You must grant this role the necessary permissions so that SageMaker can successfully complete model training. StoppingCondition - To help cap training costs, use MaxRuntimeInSeconds to set a time limit for training. Use MaxWaitTimeInSeconds to specify how long a managed spot training job has to complete. Environment - The environment variables to set in the Docker container. Do not include any security-sensitive information including account access IDs, secrets, or tokens in any environment fields. As part of the shared responsibility model, you are responsible for any potential exposure, unauthorized access, or compromise of your sensitive data if caused by security-sensitive information included in the request environment variable or plain text fields. RetryStrategy - The number of times to retry the job when the job fails due to an InternalServerError. For more information about SageMaker, see How It Works.

INSERT INTO aws.sagemaker.training_jobs (
TrainingJobName,
HyperParameters,
AlgorithmSpecification,
RoleArn,
InputDataConfig,
OutputDataConfig,
ResourceConfig,
VpcConfig,
StoppingCondition,
Tags,
EnableNetworkIsolation,
EnableInterContainerTrafficEncryption,
EnableManagedSpotTraining,
CheckpointConfig,
DebugHookConfig,
DebugRuleConfigurations,
TensorBoardOutputConfig,
ExperimentConfig,
ProfilerConfig,
ProfilerRuleConfigurations,
Environment,
RetryStrategy,
RemoteDebugConfig,
InfraCheckConfig,
SessionChainingConfig,
ServerlessJobConfig,
MlflowConfig,
ModelPackageConfig,
region
)
SELECT 
'{{ TrainingJobName }}' /* required */,
'{{ HyperParameters }}',
'{{ AlgorithmSpecification }}',
'{{ RoleArn }}' /* required */,
'{{ InputDataConfig }}',
'{{ OutputDataConfig }}' /* required */,
'{{ ResourceConfig }}',
'{{ VpcConfig }}',
'{{ StoppingCondition }}',
'{{ Tags }}',
{{ EnableNetworkIsolation }},
{{ EnableInterContainerTrafficEncryption }},
{{ EnableManagedSpotTraining }},
'{{ CheckpointConfig }}',
'{{ DebugHookConfig }}',
'{{ DebugRuleConfigurations }}',
'{{ TensorBoardOutputConfig }}',
'{{ ExperimentConfig }}',
'{{ ProfilerConfig }}',
'{{ ProfilerRuleConfigurations }}',
'{{ Environment }}',
'{{ RetryStrategy }}',
'{{ RemoteDebugConfig }}',
'{{ InfraCheckConfig }}',
'{{ SessionChainingConfig }}',
'{{ ServerlessJobConfig }}',
'{{ MlflowConfig }}',
'{{ ModelPackageConfig }}',
'{{ region }}'
RETURNING
training_job_arn
;

UPDATE examples​

Update a model training job to request a new Debugger profiling configuration or to change warm pool retention length.

UPDATE aws.sagemaker.training_jobs
SET 
TrainingJobName = '{{ TrainingJobName }}',
ProfilerConfig = '{{ ProfilerConfig }}',
ProfilerRuleConfigurations = '{{ ProfilerRuleConfigurations }}',
ResourceConfig = '{{ ResourceConfig }}',
RemoteDebugConfig = '{{ RemoteDebugConfig }}'
WHERE 
region = '{{ region }}' --required
AND TrainingJobName = '{{ TrainingJobName }}' --required
RETURNING
training_job_arn;

DELETE examples​

Deletes a training job. After SageMaker deletes a training job, all of the metadata for the training job is lost. You can delete only training jobs that are in a terminal state (Stopped, Failed, or Completed) and don't retain an Available managed warm pool. You cannot delete a job that is in the InProgress or Stopping state. After deleting the job, you can reuse its name to create another training job.

DELETE FROM aws.sagemaker.training_jobs
WHERE region = '{{ region }}' --required
;

Lifecycle Methods​

Stops a training job. To stop a job, SageMaker sends the algorithm the SIGTERM signal, which delays job termination for 120 seconds. Algorithms might use this 120-second window to save the model artifacts, so the results of the training is not lost. When it receives a StopTrainingJob request, SageMaker changes the status of the job to Stopping. After SageMaker stops the job, it sets the status to Stopped.

EXEC aws.sagemaker.training_jobs.stop_training_job 
@region='{{ region }}' --required 
@@json=
'{
"TrainingJobName": "{{ TrainingJobName }}"
}'
;
Last updated on