schemas

Creates, updates, deletes, gets or lists a schemas resource.

Overview

Name	`schemas`
Type	Resource
Id	`aws.glue.schemas`

Fields

The following fields are returned by SELECT queries:

get_schema
list_schemas

Name	Datatype	Description
`compatibility`	`string`	The compatibility mode of the schema. (NONE, DISABLED, BACKWARD, BACKWARD_ALL, FORWARD, FORWARD_ALL, FULL, FULL_ALL)
`created_time`	`string`	The date and time the schema was created.
`data_format`	`string`	The data format of the schema definition. Currently AVRO, JSON and PROTOBUF are supported. (AVRO, JSON, PROTOBUF)
`description`	`string`	A description of schema if specified when created (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>)
`latest_schema_version`	`integer (int64)`	The latest version of the schema associated with the returned schema definition.
`next_schema_version`	`integer (int64)`	The next version of the schema associated with the returned schema definition.
`registry_arn`	`string`	The Amazon Resource Name (ARN) of the registry. (pattern: <code>arn:aws(-(cn\|us-gov\|iso(-[bef])?))?:glue:.*</code>)
`registry_name`	`string`	The name of the registry. (pattern: <code>[a-zA-Z0-9-_$#.]+</code>)
`schema_arn`	`string`	The Amazon Resource Name (ARN) of the schema. (pattern: <code>arn:aws(-(cn\|us-gov\|iso(-[bef])?))?:glue:.*</code>)
`schema_checkpoint`	`integer (int64)`	The version number of the checkpoint (the last time the compatibility mode was changed).
`schema_name`	`string`	The name of the schema. (pattern: <code>[a-zA-Z0-9-_$#.]+</code>)
`schema_status`	`string`	The status of the schema. (AVAILABLE, PENDING, DELETING)
`updated_time`	`string`	The date and time the schema was updated.

Name	Datatype	Description
`created_time`	`string`	The date and time that a schema was created.
`description`	`string`	A description for the schema. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>)
`registry_name`	`string`	the name of the registry where the schema resides. (pattern: <code>[a-zA-Z0-9-_$#.]+</code>)
`schema_arn`	`string`	The Amazon Resource Name (ARN) for the schema. (pattern: <code>arn:aws(-(cn\|us-gov\|iso(-[bef])?))?:glue:.*</code>)
`schema_name`	`string`	The name of the schema. (pattern: <code>[a-zA-Z0-9-_$#.]+</code>)
`schema_status`	`string`	The status of the schema. (AVAILABLE, PENDING, DELETING)
`updated_time`	`string`	The date and time that a schema was updated.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Description
`get_schema`	`select`	`region`	Describes the specified schema in detail.
`list_schemas`	`select`	`region`	Returns a list of schemas with minimal details. Schemas in Deleting status will not be included in the results. Empty results will be returned if there are no schemas available. When the RegistryId is not provided, all the schemas across registries will be part of the API response.
`create_schema`	`insert`	`region`, `SchemaName`, `DataFormat`	Creates a new schema set and registers the schema definition. Returns an error if the schema set already exists without actually registering the version. When the schema set is created, a version checkpoint will be set to the first version. Compatibility mode "DISABLED" restricts any additional schema versions from being added after the first schema version. For all other compatibility modes, validation of compatibility settings will be applied only from the second version onwards when the RegisterSchemaVersion API is used. When this API is called without a RegistryId, this will create an entry for a "default-registry" in the registry database tables, if it is not already present.
`update_schema`	`update`	`region`, `SchemaId`	Updates the description, compatibility setting, or version checkpoint for a schema set. For updating the compatibility setting, the call will not validate compatibility for the entire set of schema versions with the new compatibility setting. If the value for Compatibility is provided, the VersionNumber (a checkpoint) is also required. The API will validate the checkpoint version number for consistency. If the value for the VersionNumber (checkpoint) is provided, Compatibility is optional and this can be used to set/reset a checkpoint for the schema. This update will happen only if the schema is in the AVAILABLE state.
`delete_schema`	`delete`	`region`	Deletes the entire schema set, including the schema set and all of its versions. To get the status of the delete operation, you can call GetSchema API after the asynchronous call. Deleting a registry will deactivate all online operations for the schema, such as the GetSchemaByDefinition, and RegisterSchemaVersion APIs.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
`region`	`string`	AWS region (default: us-east-1)

`SELECT` examples

get_schema
list_schemas

Describes the specified schema in detail.

SELECT
compatibility,
created_time,
data_format,
description,
latest_schema_version,
next_schema_version,
registry_arn,
registry_name,
schema_arn,
schema_checkpoint,
schema_name,
schema_status,
updated_time
FROM aws.glue.schemas
WHERE region = '{{ region }}' -- required
;

Returns a list of schemas with minimal details. Schemas in Deleting status will not be included in the results. Empty results will be returned if there are no schemas available. When the RegistryId is not provided, all the schemas across registries will be part of the API response.

SELECT
created_time,
description,
registry_name,
schema_arn,
schema_name,
schema_status,
updated_time
FROM aws.glue.schemas
WHERE region = '{{ region }}' -- required
;

`INSERT` examples

create_schema
Manifest

Creates a new schema set and registers the schema definition. Returns an error if the schema set already exists without actually registering the version. When the schema set is created, a version checkpoint will be set to the first version. Compatibility mode "DISABLED" restricts any additional schema versions from being added after the first schema version. For all other compatibility modes, validation of compatibility settings will be applied only from the second version onwards when the RegisterSchemaVersion API is used. When this API is called without a RegistryId, this will create an entry for a "default-registry" in the registry database tables, if it is not already present.

INSERT INTO aws.glue.schemas (
RegistryId,
SchemaName,
DataFormat,
Compatibility,
Description,
Tags,
SchemaDefinition,
region
)
SELECT 
'{{ RegistryId }}',
'{{ SchemaName }}' /* required */,
'{{ DataFormat }}' /* required */,
'{{ Compatibility }}',
'{{ Description }}',
'{{ Tags }}',
'{{ SchemaDefinition }}',
'{{ region }}'
RETURNING
compatibility,
data_format,
description,
latest_schema_version,
next_schema_version,
registry_arn,
registry_name,
schema_arn,
schema_checkpoint,
schema_name,
schema_status,
schema_version_id,
schema_version_status,
tags
;

# Description fields are for documentation purposes
- name: schemas
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the schemas resource.
  - name: RegistryId
    description: |
      This is a wrapper shape to contain the registry identity fields. If this is not provided, the default registry will be used. The ARN format for the same will be: arn:aws:glue:us-east-2:<customer id>:registry/default-registry:random-5-letter-id.
    value:
      RegistryName: "{{ RegistryName }}"
      RegistryArn: "{{ RegistryArn }}"
  - name: SchemaName
    value: "{{ SchemaName }}"
    description: |
      Name of the schema to be created of max length of 255, and may only contain letters, numbers, hyphen, underscore, dollar sign, or hash mark. No whitespace.
  - name: DataFormat
    value: "{{ DataFormat }}"
    description: |
      The data format of the schema definition. Currently AVRO, JSON and PROTOBUF are supported.
    valid_values: ['AVRO', 'JSON', 'PROTOBUF']
  - name: Compatibility
    value: "{{ Compatibility }}"
    description: |
      The compatibility mode of the schema. The possible values are: NONE: No compatibility mode applies. You can use this choice in development scenarios or if you do not know the compatibility mode that you want to apply to schemas. Any new version added will be accepted without undergoing a compatibility check. DISABLED: This compatibility choice prevents versioning for a particular schema. You can use this choice to prevent future versioning of a schema. BACKWARD: This compatibility choice is recommended as it allows data receivers to read both the current and one previous schema version. This means that for instance, a new schema version cannot drop data fields or change the type of these fields, so they can't be read by readers using the previous version. BACKWARD_ALL: This compatibility choice allows data receivers to read both the current and all previous schema versions. You can use this choice when you need to delete fields or add optional fields, and check compatibility against all previous schema versions. FORWARD: This compatibility choice allows data receivers to read both the current and one next schema version, but not necessarily later versions. You can use this choice when you need to add fields or delete optional fields, but only check compatibility against the last schema version. FORWARD_ALL: This compatibility choice allows data receivers to read written by producers of any new registered schema. You can use this choice when you need to add fields or delete optional fields, and check compatibility against all previous schema versions. FULL: This compatibility choice allows data receivers to read data written by producers using the previous or next version of the schema, but not necessarily earlier or later versions. You can use this choice when you need to add or remove optional fields, but only check compatibility against the last schema version. FULL_ALL: This compatibility choice allows data receivers to read data written by producers using all previous schema versions. You can use this choice when you need to add or remove optional fields, and check compatibility against all previous schema versions.
    valid_values: ['NONE', 'DISABLED', 'BACKWARD', 'BACKWARD_ALL', 'FORWARD', 'FORWARD_ALL', 'FULL', 'FULL_ALL']
  - name: Description
    value: "{{ Description }}"
    description: |
      An optional description of the schema. If description is not provided, there will not be any automatic default value for this.
  - name: Tags
    value: "{{ Tags }}"
    description: |
      Amazon Web Services tags that contain a key value pair and may be searched by console, command line, or API. If specified, follows the Amazon Web Services tags-on-create pattern.
  - name: SchemaDefinition
    value: "{{ SchemaDefinition }}"
    description: |
      The schema definition using the DataFormat setting for SchemaName.

`UPDATE` examples

update_schema

Updates the description, compatibility setting, or version checkpoint for a schema set. For updating the compatibility setting, the call will not validate compatibility for the entire set of schema versions with the new compatibility setting. If the value for Compatibility is provided, the VersionNumber (a checkpoint) is also required. The API will validate the checkpoint version number for consistency. If the value for the VersionNumber (checkpoint) is provided, Compatibility is optional and this can be used to set/reset a checkpoint for the schema. This update will happen only if the schema is in the AVAILABLE state.

UPDATE aws.glue.schemas
SET 
SchemaId = '{{ SchemaId }}',
SchemaVersionNumber = '{{ SchemaVersionNumber }}',
Compatibility = '{{ Compatibility }}',
Description = '{{ Description }}'
WHERE 
region = '{{ region }}' --required
AND SchemaId = '{{ SchemaId }}' --required
RETURNING
registry_name,
schema_arn,
schema_name;

`DELETE` examples

delete_schema

Deletes the entire schema set, including the schema set and all of its versions. To get the status of the delete operation, you can call GetSchema API after the asynchronous call. Deleting a registry will deactivate all online operations for the schema, such as the GetSchemaByDefinition, and RegisterSchemaVersion APIs.

DELETE FROM aws.glue.schemas
WHERE region = '{{ region }}' --required
;

Overview​

Fields​

Methods​

Parameters​

SELECT examples​

INSERT examples​

UPDATE examples​

DELETE examples​