Skip to main content

ml_transforms

Creates, updates, deletes, gets or lists a ml_transforms resource.

Overview​

Nameml_transforms
TypeResource
Idaws.glue.ml_transforms

Fields​

The following fields are returned by SELECT queries:

NameDatatypeDescription
created_onstring (date-time)The date and time when the transform was created.
descriptionstringA description of the transform. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>)
evaluation_metricsobjectThe latest evaluation metrics.
glue_versionstringThis value determines which version of Glue this machine learning transform is compatible with. Glue 1.0 is recommended for most customers. If the value is not set, the Glue compatibility defaults to Glue 0.9. For more information, see Glue Versions in the developer guide. (pattern: <code>^(\w+.)+\w+$</code>)
input_record_tablesarrayA list of Glue table definitions used by the transform.
label_countintegerThe number of labels available for this transform.
last_modified_onstring (date-time)The date and time when the transform was last modified.
max_capacitynumber (double)The number of Glue data processing units (DPUs) that are allocated to task runs for this transform. You can allocate from 2 to 100 DPUs; the default is 10. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory. For more information, see the Glue pricing page. When the WorkerType field is set to a value other than Standard, the MaxCapacity field is set automatically and becomes read-only.
max_retriesintegerThe maximum number of times to retry a task for this transform after a task run fails.
namestringThe unique name given to the transform when it was created. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>)
number_of_workersintegerThe number of workers of a defined workerType that are allocated when this task runs.
parametersobjectThe configuration parameters that are specific to the algorithm used.
rolestringThe name or Amazon Resource Name (ARN) of the IAM role with the required permissions.
schemaarrayThe Map<Column, Type> object that represents the schema that this transform accepts. Has an upper bound of 100 columns.
statusstringThe last known status of the transform (to indicate whether it can be used or not). One of "NOT_READY", "READY", or "DELETING". (NOT_READY, READY, DELETING)
timeoutintegerThe timeout for a task run for this transform in minutes. This is the maximum time that a task run for this transform can consume resources before it is terminated and enters TIMEOUT status. The default is 2,880 minutes (48 hours).
transform_encryptionobjectThe encryption-at-rest settings of the transform that apply to accessing user data. Machine learning transforms can access user data encrypted in Amazon S3 using KMS.
transform_idstringThe unique identifier of the transform, generated at the time that the transform was created. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>)
worker_typestringThe type of predefined worker that is allocated when this task runs. Accepts a value of Standard, G.1X, or G.2X. For the Standard worker type, each worker provides 4 vCPU, 16 GB of memory and a 50GB disk, and 2 executors per worker. For the G.1X worker type, each worker provides 4 vCPU, 16 GB of memory and a 64GB disk, and 1 executor per worker. For the G.2X worker type, each worker provides 8 vCPU, 32 GB of memory and a 128GB disk, and 1 executor per worker. (Standard, G.1X, G.2X, G.025X, G.4X, G.8X, Z.2X)

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
get_ml_transformselectregionGets an Glue machine learning transform artifact and all its corresponding metadata. Machine learning transforms are a special type of transform that use machine learning to learn the details of the transformation to be performed by learning from examples provided by humans. These transformations are then saved by Glue. You can retrieve their metadata by calling GetMLTransform.
get_ml_transformsselectregionGets a sortable, filterable list of existing Glue machine learning transforms. Machine learning transforms are a special type of transform that use machine learning to learn the details of the transformation to be performed by learning from examples provided by humans. These transformations are then saved by Glue, and you can retrieve their metadata by calling GetMLTransforms.
create_ml_transforminsertregion, InputRecordTablesCreates an Glue machine learning transform. This operation creates the transform and all the necessary parameters to train it. Call this operation as the first step in the process of using a machine learning transform (such as the FindMatches transform) for deduplicating data. You can provide an optional Description, in addition to the parameters that you want to use for your algorithm. You must also specify certain parameters for the tasks that Glue runs on your behalf as part of learning from your data and creating a high-quality machine learning transform. These parameters include Role, and optionally, AllocatedCapacity, Timeout, and MaxRetries. For more information, see Jobs.
update_ml_transformupdateregion, TransformIdUpdates an existing machine learning transform. Call this operation to tune the algorithm parameters to achieve better results. After calling this operation, you can call the StartMLEvaluationTaskRun operation to assess how well your new parameters achieved your goals (such as improving the quality of your machine learning transform, or making it more cost-effective).
delete_ml_transformdeleteregionDeletes an Glue machine learning transform. Machine learning transforms are a special type of transform that use machine learning to learn the details of the transformation to be performed by learning from examples provided by humans. These transformations are then saved by Glue. If you no longer need a transform, you can delete it by calling DeleteMLTransforms. However, any Glue jobs that still reference the deleted transform will no longer succeed.
list_ml_transformsexecregionRetrieves a sortable, filterable list of existing Glue machine learning transforms in this Amazon Web Services account, or the resources with the specified tag. This operation takes the optional Tags field, which you can use as a filter of the responses so that tagged resources can be retrieved as a group. If you choose to use tag filtering, only resources with the tags are retrieved.

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
regionstringAWS region (default: us-east-1)

SELECT examples​

Gets an Glue machine learning transform artifact and all its corresponding metadata. Machine learning transforms are a special type of transform that use machine learning to learn the details of the transformation to be performed by learning from examples provided by humans. These transformations are then saved by Glue. You can retrieve their metadata by calling GetMLTransform.

SELECT
created_on,
description,
evaluation_metrics,
glue_version,
input_record_tables,
label_count,
last_modified_on,
max_capacity,
max_retries,
name,
number_of_workers,
parameters,
role,
schema,
status,
timeout,
transform_encryption,
transform_id,
worker_type
FROM aws.glue.ml_transforms
WHERE region = '{{ region }}' -- required
;

INSERT examples​

Creates an Glue machine learning transform. This operation creates the transform and all the necessary parameters to train it. Call this operation as the first step in the process of using a machine learning transform (such as the FindMatches transform) for deduplicating data. You can provide an optional Description, in addition to the parameters that you want to use for your algorithm. You must also specify certain parameters for the tasks that Glue runs on your behalf as part of learning from your data and creating a high-quality machine learning transform. These parameters include Role, and optionally, AllocatedCapacity, Timeout, and MaxRetries. For more information, see Jobs.

INSERT INTO aws.glue.ml_transforms (
Name,
Description,
InputRecordTables,
Parameters,
Role,
GlueVersion,
MaxCapacity,
WorkerType,
NumberOfWorkers,
Timeout,
MaxRetries,
Tags,
TransformEncryption,
region
)
SELECT 
'{{ Name }}',
'{{ Description }}',
'{{ InputRecordTables }}' /* required */,
'{{ Parameters }}',
'{{ Role }}',
'{{ GlueVersion }}',
{{ MaxCapacity }},
'{{ WorkerType }}',
{{ NumberOfWorkers }},
{{ Timeout }},
{{ MaxRetries }},
'{{ Tags }}',
'{{ TransformEncryption }}',
'{{ region }}'
RETURNING
transform_id
;

UPDATE examples​

Updates an existing machine learning transform. Call this operation to tune the algorithm parameters to achieve better results. After calling this operation, you can call the StartMLEvaluationTaskRun operation to assess how well your new parameters achieved your goals (such as improving the quality of your machine learning transform, or making it more cost-effective).

UPDATE aws.glue.ml_transforms
SET 
TransformId = '{{ TransformId }}',
Name = '{{ Name }}',
Description = '{{ Description }}',
Parameters = '{{ Parameters }}',
Role = '{{ Role }}',
GlueVersion = '{{ GlueVersion }}',
MaxCapacity = {{ MaxCapacity }},
WorkerType = '{{ WorkerType }}',
NumberOfWorkers = {{ NumberOfWorkers }},
Timeout = {{ Timeout }},
MaxRetries = {{ MaxRetries }}
WHERE 
region = '{{ region }}' --required
AND TransformId = '{{ TransformId }}' --required
RETURNING
transform_id;

DELETE examples​

Deletes an Glue machine learning transform. Machine learning transforms are a special type of transform that use machine learning to learn the details of the transformation to be performed by learning from examples provided by humans. These transformations are then saved by Glue. If you no longer need a transform, you can delete it by calling DeleteMLTransforms. However, any Glue jobs that still reference the deleted transform will no longer succeed.

DELETE FROM aws.glue.ml_transforms
WHERE region = '{{ region }}' --required
;

Lifecycle Methods​

Retrieves a sortable, filterable list of existing Glue machine learning transforms in this Amazon Web Services account, or the resources with the specified tag. This operation takes the optional Tags field, which you can use as a filter of the responses so that tagged resources can be retrieved as a group. If you choose to use tag filtering, only resources with the tags are retrieved.

EXEC aws.glue.ml_transforms.list_ml_transforms 
@region='{{ region }}' --required 
@@json=
'{
"NextToken": "{{ NextToken }}", 
"MaxResults": {{ MaxResults }}, 
"Filter": "{{ Filter }}", 
"Sort": "{{ Sort }}", 
"Tags": "{{ Tags }}"
}'
;
Last updated on