catalogs

Creates, updates, deletes, gets or lists a catalogs resource.

Overview

Name	catalogs
Type	Resource
Id	aws.glue.catalogs

Fields

The following fields are returned by SELECT queries:

get_catalog

Name	Datatype	Description
allow_full_table_external_data_access	string	Allows third-party engines to access data in Amazon S3 locations that are registered with Lake Formation. (True, False)
catalog_id	string	The ID of the catalog. To grant access to the default catalog, this field should not be provided. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>)
catalog_properties	object	A CatalogProperties object that specifies data lake access properties and other custom properties.
create_database_default_permissions	array	An array of PrincipalPermissions objects. Creates a set of default permissions on the database(s) for principals. Used by Amazon Web Services Lake Formation. Not used in the normal course of Glue operations.
create_table_default_permissions	array	An array of PrincipalPermissions objects. Creates a set of default permissions on the table(s) for principals. Used by Amazon Web Services Lake Formation. Not used in the normal course of Glue operations.
create_time	string (date-time)	The time at which the catalog was created.
description	string	Description string, not more than 2048 bytes long, matching the URI address multi-line string pattern. A description of the catalog. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>)
federated_catalog	object	A FederatedCatalog object that points to an entity outside the Glue Data Catalog.
name	string	The name of the catalog. Cannot be the same as the account ID. (pattern: <code>^(?!(.[./\]|aws:)).$</code>)
parameters	object	A map array of key-value pairs that define parameters and properties of the catalog.
resource_arn	string	The Amazon Resource Name (ARN) assigned to the catalog resource.
target_redshift_catalog	object	A TargetRedshiftCatalog object that describes a target catalog for database resource linking.
update_time	string (date-time)	The time at which the catalog was last updated.

get_catalogs

Name	Datatype	Description
catalog_list	array	An array of Catalog objects. A list of Catalog objects from the specified parent catalog.
next_token	string	A continuation token for paginating the returned list of tokens, returned if the current segment of the list is not the last.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_catalog	select	region		The name of the Catalog to retrieve. This should be all lowercase.
get_catalogs	select	region		Retrieves all catalogs defined in a catalog in the Glue Data Catalog. For a Redshift-federated catalog use case, this operation returns the list of catalogs mapped to Redshift databases in the Redshift namespace catalog.
create_catalog	insert	region, Name, CatalogInput		Creates a new catalog in the Glue Data Catalog.
update_catalog	update	region, CatalogId, CatalogInput		Updates an existing catalog's properties in the Glue Data Catalog.
delete_catalog	delete	region		Removes the specified catalog from the Glue Data Catalog. After completing this operation, you no longer have access to the databases, tables (and all table versions and partitions that might belong to the tables) and the user-defined functions in the deleted catalog. Glue deletes these "orphaned" resources asynchronously in a timely manner, at the discretion of the service. To ensure the immediate deletion of all related resources before calling the DeleteCatalog operation, use DeleteTableVersion (or BatchDeleteTableVersion), DeletePartition (or BatchDeletePartition), DeleteTable (or BatchDeleteTable), DeleteUserDefinedFunction and DeleteDatabase to delete any resources that belong to the catalog.
import_catalog_to_glue	exec	region		Imports an existing Amazon Athena Data Catalog to Glue.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
region	string	AWS region (default: us-east-1)

SELECT examples

get_catalog

The name of the Catalog to retrieve. This should be all lowercase.

SELECT
allow_full_table_external_data_access,
catalog_id,
catalog_properties,
create_database_default_permissions,
create_table_default_permissions,
create_time,
description,
federated_catalog,
name,
parameters,
resource_arn,
target_redshift_catalog,
update_time
FROM aws.glue.catalogs
WHERE region = '{{ region }}' -- required
;

get_catalogs

Retrieves all catalogs defined in a catalog in the Glue Data Catalog. For a Redshift-federated catalog use case, this operation returns the list of catalogs mapped to Redshift databases in the Redshift namespace catalog.

SELECT
catalog_list,
next_token
FROM aws.glue.catalogs
WHERE region = '{{ region }}' -- required
;

INSERT examples

create_catalog

Creates a new catalog in the Glue Data Catalog.

INSERT INTO aws.glue.catalogs (
Name,
CatalogInput,
Tags,
region
)
SELECT 
'{{ Name }}' /* required */,
'{{ CatalogInput }}' /* required */,
'{{ Tags }}',
'{{ region }}'
;

Manifest

# Description fields are for documentation purposes
- name: catalogs
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the catalogs resource.
  - name: Name
    value: "{{ Name }}"
    description: |
      The name of the catalog to create.
  - name: CatalogInput
    description: |
      A CatalogInput object that defines the metadata for the catalog.
    value:
      Description: "{{ Description }}"
      FederatedCatalog:
        Identifier: "{{ Identifier }}"
        ConnectionName: "{{ ConnectionName }}"
        ConnectionType: "{{ ConnectionType }}"
      Parameters: "{{ Parameters }}"
      TargetRedshiftCatalog:
        CatalogArn: "{{ CatalogArn }}"
      CatalogProperties:
        DataLakeAccessProperties:
          DataLakeAccess: {{ DataLakeAccess }}
          DataTransferRole: "{{ DataTransferRole }}"
          KmsKey: "{{ KmsKey }}"
          CatalogType: "{{ CatalogType }}"
        IcebergOptimizationProperties:
          RoleArn: "{{ RoleArn }}"
          Compaction: "{{ Compaction }}"
          Retention: "{{ Retention }}"
          OrphanFileDeletion: "{{ OrphanFileDeletion }}"
        CustomProperties: "{{ CustomProperties }}"
      CreateTableDefaultPermissions:
        - Principal:
            DataLakePrincipalIdentifier: "{{ DataLakePrincipalIdentifier }}"
          Permissions: "{{ Permissions }}"
      CreateDatabaseDefaultPermissions:
        - Principal:
            DataLakePrincipalIdentifier: "{{ DataLakePrincipalIdentifier }}"
          Permissions: "{{ Permissions }}"
      AllowFullTableExternalDataAccess: "{{ AllowFullTableExternalDataAccess }}"
      OverwriteChildResourcePermissionsWithDefault: "{{ OverwriteChildResourcePermissionsWithDefault }}"
  - name: Tags
    value: "{{ Tags }}"
    description: |
      A map array of key-value pairs, not more than 50 pairs. Each key is a UTF-8 string, not less than 1 or more than 128 bytes long. Each value is a UTF-8 string, not more than 256 bytes long. The tags you assign to the catalog.

UPDATE examples

update_catalog

Updates an existing catalog's properties in the Glue Data Catalog.

UPDATE aws.glue.catalogs
SET 
CatalogId = '{{ CatalogId }}',
CatalogInput = '{{ CatalogInput }}'
WHERE 
region = '{{ region }}' --required
AND CatalogId = '{{ CatalogId }}' --required
AND CatalogInput = '{{ CatalogInput }}' --required;

DELETE examples

delete_catalog

Removes the specified catalog from the Glue Data Catalog. After completing this operation, you no longer have access to the databases, tables (and all table versions and partitions that might belong to the tables) and the user-defined functions in the deleted catalog. Glue deletes these "orphaned" resources asynchronously in a timely manner, at the discretion of the service. To ensure the immediate deletion of all related resources before calling the DeleteCatalog operation, use DeleteTableVersion (or BatchDeleteTableVersion), DeletePartition (or BatchDeletePartition), DeleteTable (or BatchDeleteTable), DeleteUserDefinedFunction and DeleteDatabase to delete any resources that belong to the catalog.

DELETE FROM aws.glue.catalogs
WHERE region = '{{ region }}' --required
;

Lifecycle Methods

import_catalog_to_glue

Imports an existing Amazon Athena Data Catalog to Glue.

EXEC aws.glue.catalogs.import_catalog_to_glue 
@region='{{ region }}' --required 
@@json=
'{
"CatalogId": "{{ CatalogId }}"
}'
;