data_catalogs

Creates, updates, deletes, gets or lists a data_catalogs resource.

Overview

Name	`data_catalogs`
Type	Resource
Id	`aws.athena.data_catalogs`

Fields

The following fields are returned by SELECT queries:

get_data_catalog
list_data_catalogs

Name	Datatype	Description
`connection_type`	`string`	The type of connection for a FEDERATED data catalog (for example, REDSHIFT, MYSQL, or SQLSERVER). For information about individual connectors, see Available data source connectors. (DYNAMODB, MYSQL, POSTGRESQL, REDSHIFT, ORACLE, SYNAPSE, SQLSERVER, DB2, OPENSEARCH, BIGQUERY, GOOGLECLOUDSTORAGE, HBASE, DOCUMENTDB, CMDB, TPCDS, TIMESTREAM, SAPHANA, SNOWFLAKE, DATALAKEGEN2, DB2AS400)
`description`	`string`	An optional description of the data catalog.
`error`	`string`	Text of the error that occurred during data catalog creation or deletion.
`name`	`string`	The name of the data catalog. The catalog name must be unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>)
`parameters`	`object`	Specifies the Lambda function or functions to use for the data catalog. This is a mapping whose values depend on the catalog type. For the HIVE data catalog type, use the following syntax. The metadata-function parameter is required. The sdk-version parameter is optional and defaults to the currently supported version. metadata-function=lambda_arn, sdk-version=version_number For the LAMBDA data catalog type, use one of the following sets of required parameters, but not both. If you have one Lambda function that processes metadata and another for reading the actual data, use the following syntax. Both parameters are required. metadata-function=lambda_arn, record-function=lambda_arn If you have a composite Lambda function that processes both metadata and data, use the following syntax to specify your Lambda function. function=lambda_arn The GLUE type takes a catalog ID parameter and is required. The catalog_id is the account ID of the Amazon Web Services account to which the Glue catalog belongs. catalog-id=catalog_id The GLUE data catalog type also applies to the default AwsDataCatalog that already exists in your account, of which you can have only one and cannot modify. The FEDERATED data catalog type uses one of the following parameters, but not both. Use connection-arn for an existing Glue connection. Use connection-type and connection-properties to specify the configuration setting for a new connection. connection-arn:`<glue_connection_arn_to_reuse>` connection-type:MYSQL\|REDSHIFT\|...., connection-properties:"`<json_string>`" For `<json_string>` , use escaped JSON text, as in the following example. "{"spill_bucket":"my_spill","spill_prefix":"athena-spill","host":"abc12345.snowflakecomputing.com","port":"1234","warehouse":"DEV_WH","database":"TEST","schema":"PUBLIC","SecretArn":"arn:aws:secretsmanager:ap-south-1:111122223333:secret:snowflake-XHb67j"}"
`status`	`string`	The status of the creation or deletion of the data catalog. The LAMBDA, GLUE, and HIVE data catalog types are created synchronously. Their status is either CREATE_COMPLETE or CREATE_FAILED. The FEDERATED data catalog type is created asynchronously. Data catalog creation status: CREATE_IN_PROGRESS: Federated data catalog creation in progress. CREATE_COMPLETE: Data catalog creation complete. CREATE_FAILED: Data catalog could not be created. CREATE_FAILED_CLEANUP_IN_PROGRESS: Federated data catalog creation failed and is being removed. CREATE_FAILED_CLEANUP_COMPLETE: Federated data catalog creation failed and was removed. CREATE_FAILED_CLEANUP_FAILED: Federated data catalog creation failed but could not be removed. Data catalog deletion status: DELETE_IN_PROGRESS: Federated data catalog deletion in progress. DELETE_COMPLETE: Federated data catalog deleted. DELETE_FAILED: Federated data catalog could not be deleted. (CREATE_IN_PROGRESS, CREATE_COMPLETE, CREATE_FAILED, CREATE_FAILED_CLEANUP_IN_PROGRESS, CREATE_FAILED_CLEANUP_COMPLETE, CREATE_FAILED_CLEANUP_FAILED, DELETE_IN_PROGRESS, DELETE_COMPLETE, DELETE_FAILED)
`type`	`string`	The type of data catalog to create: LAMBDA for a federated catalog, GLUE for an Glue Data Catalog, and HIVE for an external Apache Hive metastore. FEDERATED is a federated catalog for which Athena creates the connection and the Lambda function for you based on the parameters that you pass. (LAMBDA, GLUE, HIVE, FEDERATED)

Name	Datatype	Description
`catalog_name`	`string`	The name of the data catalog. The catalog name is unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>)
`connection_type`	`string`	The type of connection for a FEDERATED data catalog (for example, REDSHIFT, MYSQL, or SQLSERVER). For information about individual connectors, see Available data source connectors. (DYNAMODB, MYSQL, POSTGRESQL, REDSHIFT, ORACLE, SYNAPSE, SQLSERVER, DB2, OPENSEARCH, BIGQUERY, GOOGLECLOUDSTORAGE, HBASE, DOCUMENTDB, CMDB, TPCDS, TIMESTREAM, SAPHANA, SNOWFLAKE, DATALAKEGEN2, DB2AS400)
`error`	`string`	Text of the error that occurred during data catalog creation or deletion.
`status`	`string`	The status of the creation or deletion of the data catalog. The LAMBDA, GLUE, and HIVE data catalog types are created synchronously. Their status is either CREATE_COMPLETE or CREATE_FAILED. The FEDERATED data catalog type is created asynchronously. Data catalog creation status: CREATE_IN_PROGRESS: Federated data catalog creation in progress. CREATE_COMPLETE: Data catalog creation complete. CREATE_FAILED: Data catalog could not be created. CREATE_FAILED_CLEANUP_IN_PROGRESS: Federated data catalog creation failed and is being removed. CREATE_FAILED_CLEANUP_COMPLETE: Federated data catalog creation failed and was removed. CREATE_FAILED_CLEANUP_FAILED: Federated data catalog creation failed but could not be removed. Data catalog deletion status: DELETE_IN_PROGRESS: Federated data catalog deletion in progress. DELETE_COMPLETE: Federated data catalog deleted. DELETE_FAILED: Federated data catalog could not be deleted. (CREATE_IN_PROGRESS, CREATE_COMPLETE, CREATE_FAILED, CREATE_FAILED_CLEANUP_IN_PROGRESS, CREATE_FAILED_CLEANUP_COMPLETE, CREATE_FAILED_CLEANUP_FAILED, DELETE_IN_PROGRESS, DELETE_COMPLETE, DELETE_FAILED)
`type`	`string`	The data catalog type. (LAMBDA, GLUE, HIVE, FEDERATED)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Description
`get_data_catalog`	`select`	`region`	Returns the specified data catalog.
`list_data_catalogs`	`select`	`region`	Lists the data catalogs in the current Amazon Web Services account. In the Athena console, data catalogs are listed as "data sources" on the Data sources page under the Data source name column.
`create_data_catalog`	`insert`	`region`, `Name`	Creates (registers) a data catalog with the specified name and properties. Catalogs created are visible to all users of the same Amazon Web Services account. For a FEDERATED catalog, this API operation creates the following resources. CFN Stack Name with a maximum length of 128 characters and prefix athenafederatedcatalog-CATALOG_NAME_SANITIZED with length 23 characters. Lambda Function Name with a maximum length of 64 characters and prefix athenafederatedcatalog_CATALOG_NAME_SANITIZED with length 23 characters. Glue Connection Name with a maximum length of 255 characters and a prefix athenafederatedcatalog_CATALOG_NAME_SANITIZED with length 23 characters.
`update_data_catalog`	`update`	`region`, `Name`	Updates the data catalog that has the specified name.
`delete_data_catalog`	`delete`	`region`	Deletes a data catalog.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
`region`	`string`	AWS region (default: us-east-1)

`SELECT` examples

get_data_catalog
list_data_catalogs

Returns the specified data catalog.

SELECT
connection_type,
description,
error,
name,
parameters,
status,
type
FROM aws.athena.data_catalogs
WHERE region = '{{ region }}' -- required
;

Lists the data catalogs in the current Amazon Web Services account. In the Athena console, data catalogs are listed as "data sources" on the Data sources page under the Data source name column.

SELECT
catalog_name,
connection_type,
error,
status,
type
FROM aws.athena.data_catalogs
WHERE region = '{{ region }}' -- required
;

`INSERT` examples

create_data_catalog
Manifest

Creates (registers) a data catalog with the specified name and properties. Catalogs created are visible to all users of the same Amazon Web Services account. For a FEDERATED catalog, this API operation creates the following resources. CFN Stack Name with a maximum length of 128 characters and prefix athenafederatedcatalog-CATALOG_NAME_SANITIZED with length 23 characters. Lambda Function Name with a maximum length of 64 characters and prefix athenafederatedcatalog_CATALOG_NAME_SANITIZED with length 23 characters. Glue Connection Name with a maximum length of 255 characters and a prefix athenafederatedcatalog_CATALOG_NAME_SANITIZED with length 23 characters.

INSERT INTO aws.athena.data_catalogs (
Name,
Type,
Description,
Parameters,
Tags,
region
)
SELECT 
'{{ Name }}' /* required */,
'{{ Type }}',
'{{ Description }}',
'{{ Parameters }}',
'{{ Tags }}',
'{{ region }}'
RETURNING
data_catalog
;

# Description fields are for documentation purposes
- name: data_catalogs
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the data_catalogs resource.
  - name: Name
    value: "{{ Name }}"
    description: |
      The name of the data catalog to create. The catalog name must be unique for the Amazon Web Services account and can use a maximum of 127 alphanumeric, underscore, at sign, or hyphen characters. The remainder of the length constraint of 256 is reserved for use by Athena. For FEDERATED type the catalog name has following considerations and limits: The catalog name allows special characters such as _ , @ ,  , - . These characters are replaced with a hyphen (-) when creating the CFN Stack Name and with an underscore (_) when creating the Lambda Function and Glue Connection Name. The catalog name has a theoretical limit of 128 characters. However, since we use it to create other resources that allow less characters and we prepend a prefix to it, the actual catalog name limit for FEDERATED catalog is 64 - 23 = 41 characters.
  - name: Type
    value: "{{ Type }}"
    description: |
      The type of data catalog to create: LAMBDA for a federated catalog, GLUE for an Glue Data Catalog, and HIVE for an external Apache Hive metastore. FEDERATED is a federated catalog for which Athena creates the connection and the Lambda function for you based on the parameters that you pass. For FEDERATED type, we do not support IAM identity center.
    valid_values: ['LAMBDA', 'GLUE', 'HIVE', 'FEDERATED']
  - name: Description
    value: "{{ Description }}"
    description: |
      A description of the data catalog to be created.
  - name: Parameters
    value: "{{ Parameters }}"
    description: |
      Specifies the Lambda function or functions to use for creating the data catalog. This is a mapping whose values depend on the catalog type. For the HIVE data catalog type, use the following syntax. The metadata-function parameter is required. The sdk-version parameter is optional and defaults to the currently supported version. metadata-function=lambda_arn, sdk-version=version_number For the LAMBDA data catalog type, use one of the following sets of required parameters, but not both. If you have one Lambda function that processes metadata and another for reading the actual data, use the following syntax. Both parameters are required. metadata-function=lambda_arn, record-function=lambda_arn If you have a composite Lambda function that processes both metadata and data, use the following syntax to specify your Lambda function. function=lambda_arn The GLUE type takes a catalog ID parameter and is required. The catalog_id is the account ID of the Amazon Web Services account to which the Glue Data Catalog belongs. catalog-id=catalog_id The GLUE data catalog type also applies to the default AwsDataCatalog that already exists in your account, of which you can have only one and cannot modify. The FEDERATED data catalog type uses one of the following parameters, but not both. Use connection-arn for an existing Glue connection. Use connection-type and connection-properties to specify the configuration setting for a new connection. connection-arn:`<glue_connection_arn_to_reuse>` lambda-role-arn (optional): The execution role to use for the Lambda function. If not provided, one is created. connection-type:MYSQL|REDSHIFT|...., connection-properties:"`<json_string>`" For `<json_string>` , use escaped JSON text, as in the following example. "{"spill_bucket":"my_spill","spill_prefix":"athena-spill","host":"abc12345.snowflakecomputing.com","port":"1234","warehouse":"DEV_WH","database":"TEST","schema":"PUBLIC","SecretArn":"arn:aws:secretsmanager:ap-south-1:111122223333:secret:snowflake-XHb67j"}"
  - name: Tags
    description: |
      A list of comma separated tags to add to the data catalog that is created. All the resources that are created by the CreateDataCatalog API operation with FEDERATED type will have the tag federated_athena_datacatalog="true". This includes the CFN Stack, Glue Connection, Athena DataCatalog, and all the resources created as part of the CFN Stack (Lambda Function, IAM policies/roles).
    value:
      - Key: "{{ Key }}"
        Value: "{{ Value }}"

`UPDATE` examples

update_data_catalog

Updates the data catalog that has the specified name.

UPDATE aws.athena.data_catalogs
SET 
Name = '{{ Name }}',
Type = '{{ Type }}',
Description = '{{ Description }}',
Parameters = '{{ Parameters }}'
WHERE 
region = '{{ region }}' --required
AND Name = '{{ Name }}' --required;

`DELETE` examples

delete_data_catalog

Deletes a data catalog.

DELETE FROM aws.athena.data_catalogs
WHERE region = '{{ region }}' --required
;

Overview​

Fields​

Methods​

Parameters​

SELECT examples​

INSERT examples​

UPDATE examples​

DELETE examples​