processing_jobs

Creates, updates, deletes, gets or lists a processing_jobs resource.

Overview

Name	`processing_jobs`
Type	Resource
Id	`aws.sagemaker.processing_jobs`

Fields

The following fields are returned by SELECT queries:

describe_processing_job
list_processing_jobs

Name	Datatype	Description
`app_specification`	`object`	Configures the processing job to run a specified container image.
`auto_ml_job_arn`	`string`	The ARN of an AutoML job associated with this processing job. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:automl-job/.*</code>)
`creation_time`	`string (date-time)`	The time at which the processing job was created.
`environment`	`object`	The environment variables set in the Docker container.
`exit_message`	`string`	An optional string, up to one KB in size, that contains metadata from the processing container when the processing job exits. (pattern: <code>[\S\s]*</code>)
`experiment_config`	`object`	Associates a SageMaker job as a trial component with an experiment and trial. Specified when you call the following APIs: CreateProcessingJob CreateTrainingJob CreateTransformJob
`failure_reason`	`string`	A string, up to one KB in size, that contains the reason a processing job failed, if it failed.
`last_modified_time`	`string (date-time)`	The time at which the processing job was last modified.
`monitoring_schedule_arn`	`string`	The ARN of a monitoring schedule for an endpoint associated with this processing job. (pattern: <code>.*</code>)
`network_config`	`object`	Networking options for a processing job.
`processing_end_time`	`string (date-time)`	The time at which the processing job completed.
`processing_inputs`	`array`	The inputs for a processing job.
`processing_job_arn`	`string`	The Amazon Resource Name (ARN) of the processing job. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:processing-job/[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}</code>)
`processing_job_name`	`string`	The name of the processing job. The name must be unique within an Amazon Web Services Region in the Amazon Web Services account. (pattern: <code>[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}</code>)
`processing_job_status`	`string`	Provides the status of a processing job. (InProgress, Completed, Failed, Stopping, Stopped)
`processing_output_config`	`object`	Output configuration for the processing job.
`processing_resources`	`object`	Identifies the resources, ML compute instances, and ML storage volumes to deploy for a processing job. In distributed training, you specify more than one instance.
`processing_start_time`	`string (date-time)`	The time at which the processing job started.
`role_arn`	`string`	The Amazon Resource Name (ARN) of an IAM role that Amazon SageMaker can assume to perform tasks on your behalf. (pattern: <code>arn:aws[a-z-]*:iam::\d{12}:role/?[a-zA-Z_0-9+=,.@-_/]+</code>)
`stopping_condition`	`object`	The time limit for how long the processing job is allowed to run.
`training_job_arn`	`string`	The ARN of a training job associated with this processing job. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:training-job/[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}</code>)

Name	Datatype	Description
`creation_time`	`string (date-time)`	The time at which the processing job was created.
`exit_message`	`string`	An optional string, up to one KB in size, that contains metadata from the processing container when the processing job exits. (pattern: <code>[\S\s]*</code>)
`failure_reason`	`string`	A string, up to one KB in size, that contains the reason a processing job failed, if it failed.
`last_modified_time`	`string (date-time)`	A timestamp that indicates the last time the processing job was modified.
`processing_end_time`	`string (date-time)`	The time at which the processing job completed.
`processing_job_arn`	`string`	The Amazon Resource Name (ARN) of the processing job.. (pattern: <code>arn:aws[a-z-]:sagemaker:[a-z0-9-]:[0-9]{12}:processing-job/[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}</code>)
`processing_job_name`	`string`	The name of the processing job. (pattern: <code>[a-zA-Z0-9](-*[a-zA-Z0-9]){0,62}</code>)
`processing_job_status`	`string`	The status of the processing job. (InProgress, Completed, Failed, Stopping, Stopped)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Description
`describe_processing_job`	`select`	`region`	Returns a description of a processing job.
`list_processing_jobs`	`select`	`region`	Lists processing jobs that satisfy various filters.
`create_processing_job`	`insert`	`region`, `ProcessingJobName`, `ProcessingResources`, `AppSpecification`, `RoleArn`	Creates a processing job.
`delete_processing_job`	`delete`	`region`	Deletes a processing job. After Amazon SageMaker deletes a processing job, all of the metadata for the processing job is lost. You can delete only processing jobs that are in a terminal state (Stopped, Failed, or Completed). You cannot delete a job that is in the InProgress or Stopping state. After deleting the job, you can reuse its name to create another processing job.
`stop_processing_job`	`exec`	`region`, `ProcessingJobName`	Stops a processing job.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
`region`	`string`	AWS region (default: us-east-1)

`SELECT` examples

describe_processing_job
list_processing_jobs

Returns a description of a processing job.

SELECT
app_specification,
auto_ml_job_arn,
creation_time,
environment,
exit_message,
experiment_config,
failure_reason,
last_modified_time,
monitoring_schedule_arn,
network_config,
processing_end_time,
processing_inputs,
processing_job_arn,
processing_job_name,
processing_job_status,
processing_output_config,
processing_resources,
processing_start_time,
role_arn,
stopping_condition,
training_job_arn
FROM aws.sagemaker.processing_jobs
WHERE region = '{{ region }}' -- required
;

Lists processing jobs that satisfy various filters.

SELECT
creation_time,
exit_message,
failure_reason,
last_modified_time,
processing_end_time,
processing_job_arn,
processing_job_name,
processing_job_status
FROM aws.sagemaker.processing_jobs
WHERE region = '{{ region }}' -- required
;

`INSERT` examples

create_processing_job
Manifest

Creates a processing job.

INSERT INTO aws.sagemaker.processing_jobs (
ProcessingInputs,
ProcessingOutputConfig,
ProcessingJobName,
ProcessingResources,
StoppingCondition,
AppSpecification,
Environment,
NetworkConfig,
RoleArn,
Tags,
ExperimentConfig,
region
)
SELECT 
'{{ ProcessingInputs }}',
'{{ ProcessingOutputConfig }}',
'{{ ProcessingJobName }}' /* required */,
'{{ ProcessingResources }}' /* required */,
'{{ StoppingCondition }}',
'{{ AppSpecification }}' /* required */,
'{{ Environment }}',
'{{ NetworkConfig }}',
'{{ RoleArn }}' /* required */,
'{{ Tags }}',
'{{ ExperimentConfig }}',
'{{ region }}'
RETURNING
processing_job_arn
;

# Description fields are for documentation purposes
- name: processing_jobs
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the processing_jobs resource.
  - name: ProcessingInputs
    description: |
      An array of inputs configuring the data to download into the processing container.
    value:
      - InputName: "{{ InputName }}"
        AppManaged: {{ AppManaged }}
        S3Input:
          S3Uri: "{{ S3Uri }}"
          LocalPath: "{{ LocalPath }}"
          S3DataType: "{{ S3DataType }}"
          S3InputMode: "{{ S3InputMode }}"
          S3DataDistributionType: "{{ S3DataDistributionType }}"
          S3CompressionType: "{{ S3CompressionType }}"
        DatasetDefinition:
          AthenaDatasetDefinition:
            Catalog: "{{ Catalog }}"
            Database: "{{ Database }}"
            QueryString: "{{ QueryString }}"
            WorkGroup: "{{ WorkGroup }}"
            OutputS3Uri: "{{ OutputS3Uri }}"
            KmsKeyId: "{{ KmsKeyId }}"
            OutputFormat: "{{ OutputFormat }}"
            OutputCompression: "{{ OutputCompression }}"
          RedshiftDatasetDefinition:
            ClusterId: "{{ ClusterId }}"
            Database: "{{ Database }}"
            DbUser: "{{ DbUser }}"
            QueryString: "{{ QueryString }}"
            ClusterRoleArn: "{{ ClusterRoleArn }}"
            OutputS3Uri: "{{ OutputS3Uri }}"
            KmsKeyId: "{{ KmsKeyId }}"
            OutputFormat: "{{ OutputFormat }}"
            OutputCompression: "{{ OutputCompression }}"
          LocalPath: "{{ LocalPath }}"
          DataDistributionType: "{{ DataDistributionType }}"
          InputMode: "{{ InputMode }}"
  - name: ProcessingOutputConfig
    description: |
      Output configuration for the processing job.
    value:
      Outputs:
        - OutputName: "{{ OutputName }}"
          S3Output:
            S3Uri: "{{ S3Uri }}"
            LocalPath: "{{ LocalPath }}"
            S3UploadMode: "{{ S3UploadMode }}"
          FeatureStoreOutput:
            FeatureGroupName: "{{ FeatureGroupName }}"
          AppManaged: {{ AppManaged }}
      KmsKeyId: "{{ KmsKeyId }}"
  - name: ProcessingJobName
    value: "{{ ProcessingJobName }}"
    description: |
      The name of the processing job. The name must be unique within an Amazon Web Services Region in the Amazon Web Services account.
  - name: ProcessingResources
    description: |
      Identifies the resources, ML compute instances, and ML storage volumes to deploy for a processing job. In distributed training, you specify more than one instance.
    value:
      ClusterConfig:
        InstanceCount: {{ InstanceCount }}
        InstanceType: "{{ InstanceType }}"
        VolumeSizeInGB: {{ VolumeSizeInGB }}
        VolumeKmsKeyId: "{{ VolumeKmsKeyId }}"
  - name: StoppingCondition
    description: |
      The time limit for how long the processing job is allowed to run.
    value:
      MaxRuntimeInSeconds: {{ MaxRuntimeInSeconds }}
  - name: AppSpecification
    description: |
      Configures the processing job to run a specified Docker container image.
    value:
      ImageUri: "{{ ImageUri }}"
      ContainerEntrypoint:
        - "{{ ContainerEntrypoint }}"
      ContainerArguments:
        - "{{ ContainerArguments }}"
  - name: Environment
    value: "{{ Environment }}"
    description: |
      The environment variables to set in the Docker container. Up to 100 key and values entries in the map are supported. Do not include any security-sensitive information including account access IDs, secrets, or tokens in any environment fields. As part of the shared responsibility model, you are responsible for any potential exposure, unauthorized access, or compromise of your sensitive data if caused by security-sensitive information included in the request environment variable or plain text fields.
  - name: NetworkConfig
    description: |
      Networking options for a processing job, such as whether to allow inbound and outbound network calls to and from processing containers, and the VPC subnets and security groups to use for VPC-enabled processing jobs.
    value:
      EnableInterContainerTrafficEncryption: {{ EnableInterContainerTrafficEncryption }}
      EnableNetworkIsolation: {{ EnableNetworkIsolation }}
      VpcConfig:
        SecurityGroupIds:
          - "{{ SecurityGroupIds }}"
        Subnets:
          - "{{ Subnets }}"
  - name: RoleArn
    value: "{{ RoleArn }}"
    description: |
      The Amazon Resource Name (ARN) of an IAM role that Amazon SageMaker can assume to perform tasks on your behalf.
  - name: Tags
    description: |
      (Optional) An array of key-value pairs. For more information, see Using Cost Allocation Tags in the Amazon Web Services Billing and Cost Management User Guide. Do not include any security-sensitive information including account access IDs, secrets, or tokens in any tags. As part of the shared responsibility model, you are responsible for any potential exposure, unauthorized access, or compromise of your sensitive data if caused by security-sensitive information included in the request tag variable or plain text fields.
    value:
      - Key: "{{ Key }}"
        Value: "{{ Value }}"
  - name: ExperimentConfig
    description: |
      Associates a SageMaker job as a trial component with an experiment and trial. Specified when you call the following APIs: CreateProcessingJob CreateTrainingJob CreateTransformJob
    value:
      ExperimentName: "{{ ExperimentName }}"
      TrialName: "{{ TrialName }}"
      TrialComponentDisplayName: "{{ TrialComponentDisplayName }}"
      RunName: "{{ RunName }}"

`DELETE` examples

delete_processing_job

Deletes a processing job. After Amazon SageMaker deletes a processing job, all of the metadata for the processing job is lost. You can delete only processing jobs that are in a terminal state (Stopped, Failed, or Completed). You cannot delete a job that is in the InProgress or Stopping state. After deleting the job, you can reuse its name to create another processing job.

DELETE FROM aws.sagemaker.processing_jobs
WHERE region = '{{ region }}' --required
;

Lifecycle Methods

stop_processing_job

Stops a processing job.

EXEC aws.sagemaker.processing_jobs.stop_processing_job 
@region='{{ region }}' --required 
@@json=
'{
"ProcessingJobName": "{{ ProcessingJobName }}"
}'
;

Overview​

Fields​

Methods​

Parameters​

SELECT examples​

INSERT examples​

DELETE examples​

Lifecycle Methods​