crawlers

Creates, updates, deletes, gets or lists a crawlers resource.

Overview

Name	crawlers
Type	Resource
Id	aws.glue.crawlers

Fields

The following fields are returned by SELECT queries:

batch_get_crawlers

Name	Datatype	Description
crawlers	array	A list of crawler definitions.
crawlers_not_found	array	A list of names of crawlers that were not found.

get_crawler

Name	Datatype	Description
classifiers	array	A list of UTF-8 strings that specify the custom classifiers that are associated with the crawler.
configuration	string	Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Setting crawler configuration options.
crawl_elapsed_time	integer (int64)	If the crawler is running, contains the total time elapsed since the last crawl began.
crawler_security_configuration	string	The name of the SecurityConfiguration structure to be used by this crawler.
creation_time	string (date-time)	The time that the crawler was created.
database_name	string	The name of the database in which the crawler's output is stored.
description	string	A description of the crawler. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>)
lake_formation_configuration	object	Specifies whether the crawler should use Lake Formation credentials for the crawler instead of the IAM role credentials.
last_crawl	object	The status of the last crawl, and potentially error information if an error occurred.
last_updated	string (date-time)	The time that the crawler was last updated.
lineage_configuration	object	A configuration that specifies whether data lineage is enabled for the crawler.
name	string	The name of the crawler. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>)
recrawl_policy	object	A policy that specifies whether to crawl the entire dataset again, or to crawl only folders that were added since the last crawler run.
role	string	The Amazon Resource Name (ARN) of an IAM role that's used to access customer resources, such as Amazon Simple Storage Service (Amazon S3) data.
schedule	object	For scheduled crawlers, the schedule when the crawler runs.
schema_change_policy	object	The policy that specifies update and delete behaviors for the crawler.
state	string	Indicates whether the crawler is running, or whether a run is pending. (READY, RUNNING, STOPPING)
table_prefix	string	The prefix added to the names of tables that are created.
targets	object	A collection of targets to crawl.
version	integer (int64)	The version of the crawler.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
batch_get_crawlers	select	region		Returns a list of resource metadata for a given list of crawler names. After calling the ListCrawlers operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.
get_crawler	select	region		Retrieves metadata for a specified crawler.
create_crawler	insert	region		Creates a new crawler with specified targets, role, configuration, and optional schedule. At least one crawl target must be specified, in the s3Targets field, the jdbcTargets field, or the DynamoDBTargets field.
update_crawler_schedule	update	region, CrawlerName		Updates the schedule of a crawler using a cron expression.
update_crawler	update	region		Updates a crawler. If a crawler is running, you must stop it using StopCrawler before updating it.
delete_crawler	delete	region		Removes a specified crawler from the Glue Data Catalog, unless the crawler state is RUNNING.
get_crawlers	exec	region		Retrieves metadata for all crawlers defined in the customer account.
list_crawlers	exec	region		Retrieves the names of all crawler resources in this Amazon Web Services account, or the resources with the specified tag. This operation allows you to see which resources are available in your account, and their names. This operation takes the optional Tags field, which you can use as a filter on the response so that tagged resources can be retrieved as a group. If you choose to use tags filtering, only resources with the tag are retrieved.
start_crawler_schedule	exec	region, CrawlerName		Changes the schedule state of the specified crawler to SCHEDULED, unless the crawler is already running or the schedule state is already SCHEDULED.
stop_crawler_schedule	exec	region, CrawlerName		Sets the schedule state of the specified crawler to NOT_SCHEDULED, but does not stop the crawler if it is already running.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
region	string	AWS region (default: us-east-1)

SELECT examples

batch_get_crawlers

Returns a list of resource metadata for a given list of crawler names. After calling the ListCrawlers operation, you can call this operation to access the data to which you have been granted permissions. This operation supports all IAM permissions, including permission conditions that uses tags.

SELECT
crawlers,
crawlers_not_found
FROM aws.glue.crawlers
WHERE region = '{{ region }}' -- required
;

get_crawler

Retrieves metadata for a specified crawler.

SELECT
classifiers,
configuration,
crawl_elapsed_time,
crawler_security_configuration,
creation_time,
database_name,
description,
lake_formation_configuration,
last_crawl,
last_updated,
lineage_configuration,
name,
recrawl_policy,
role,
schedule,
schema_change_policy,
state,
table_prefix,
targets,
version
FROM aws.glue.crawlers
WHERE region = '{{ region }}' -- required
;

INSERT examples

create_crawler

Creates a new crawler with specified targets, role, configuration, and optional schedule. At least one crawl target must be specified, in the s3Targets field, the jdbcTargets field, or the DynamoDBTargets field.

INSERT INTO aws.glue.crawlers (
Name,
Role,
DatabaseName,
Description,
Targets,
Schedule,
Classifiers,
TablePrefix,
SchemaChangePolicy,
RecrawlPolicy,
LineageConfiguration,
LakeFormationConfiguration,
Configuration,
CrawlerSecurityConfiguration,
Tags,
region
)
SELECT 
'{{ Name }}',
'{{ Role }}',
'{{ DatabaseName }}',
'{{ Description }}',
'{{ Targets }}',
'{{ Schedule }}',
'{{ Classifiers }}',
'{{ TablePrefix }}',
'{{ SchemaChangePolicy }}',
'{{ RecrawlPolicy }}',
'{{ LineageConfiguration }}',
'{{ LakeFormationConfiguration }}',
'{{ Configuration }}',
'{{ CrawlerSecurityConfiguration }}',
'{{ Tags }}',
'{{ region }}'
;

Manifest

# Description fields are for documentation purposes
- name: crawlers
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the crawlers resource.
  - name: Name
    value: "{{ Name }}"
    description: |
      Name of the new crawler.
  - name: Role
    value: "{{ Role }}"
    description: |
      The IAM role or Amazon Resource Name (ARN) of an IAM role used by the new crawler to access customer resources.
  - name: DatabaseName
    value: "{{ DatabaseName }}"
    description: |
      The Glue database where results are written, such as: arn:aws:daylight:us-east-1::database/sometable/*.
  - name: Description
    value: "{{ Description }}"
    description: |
      A description of the new crawler.
  - name: Targets
    description: |
      A list of collection of targets to crawl.
    value:
      S3Targets:
        - Path: "{{ Path }}"
          Exclusions: "{{ Exclusions }}"
          ConnectionName: "{{ ConnectionName }}"
          SampleSize: {{ SampleSize }}
          EventQueueArn: "{{ EventQueueArn }}"
          DlqEventQueueArn: "{{ DlqEventQueueArn }}"
      JdbcTargets:
        - ConnectionName: "{{ ConnectionName }}"
          Path: "{{ Path }}"
          Exclusions: "{{ Exclusions }}"
          EnableAdditionalMetadata: "{{ EnableAdditionalMetadata }}"
      MongoDBTargets:
        - ConnectionName: "{{ ConnectionName }}"
          Path: "{{ Path }}"
          ScanAll: {{ ScanAll }}
      DynamoDBTargets:
        - Path: "{{ Path }}"
          scanAll: {{ scanAll }}
          scanRate: {{ scanRate }}
      CatalogTargets:
        - DatabaseName: "{{ DatabaseName }}"
          Tables: "{{ Tables }}"
          ConnectionName: "{{ ConnectionName }}"
          EventQueueArn: "{{ EventQueueArn }}"
          DlqEventQueueArn: "{{ DlqEventQueueArn }}"
      DeltaTargets:
        - DeltaTables: "{{ DeltaTables }}"
          ConnectionName: "{{ ConnectionName }}"
          WriteManifest: {{ WriteManifest }}
          CreateNativeDeltaTable: {{ CreateNativeDeltaTable }}
      IcebergTargets:
        - Paths: "{{ Paths }}"
          ConnectionName: "{{ ConnectionName }}"
          Exclusions: "{{ Exclusions }}"
          MaximumTraversalDepth: {{ MaximumTraversalDepth }}
      HudiTargets:
        - Paths: "{{ Paths }}"
          ConnectionName: "{{ ConnectionName }}"
          Exclusions: "{{ Exclusions }}"
          MaximumTraversalDepth: {{ MaximumTraversalDepth }}
  - name: Schedule
    value: "{{ Schedule }}"
    description: |
      A cron expression used to specify the schedule (see Time-Based Schedules for Jobs and Crawlers. For example, to run something every day at 12:15 UTC, you would specify: cron(15 12 * * ? *).
  - name: Classifiers
    value:
      - "{{ Classifiers }}"
    description: |
      A list of custom classifiers that the user has registered. By default, all built-in classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification.
  - name: TablePrefix
    value: "{{ TablePrefix }}"
    description: |
      The table prefix used for catalog tables that are created.
  - name: SchemaChangePolicy
    description: |
      The policy for the crawler's update and deletion behavior.
    value:
      UpdateBehavior: "{{ UpdateBehavior }}"
      DeleteBehavior: "{{ DeleteBehavior }}"
  - name: RecrawlPolicy
    description: |
      A policy that specifies whether to crawl the entire dataset again, or to crawl only folders that were added since the last crawler run.
    value:
      RecrawlBehavior: "{{ RecrawlBehavior }}"
  - name: LineageConfiguration
    description: |
      Specifies data lineage configuration settings for the crawler.
    value:
      CrawlerLineageSettings: "{{ CrawlerLineageSettings }}"
  - name: LakeFormationConfiguration
    description: |
      Specifies Lake Formation configuration settings for the crawler.
    value:
      UseLakeFormationCredentials: {{ UseLakeFormationCredentials }}
      AccountId: "{{ AccountId }}"
  - name: Configuration
    value: "{{ Configuration }}"
    description: |
      Crawler configuration information. This versioned JSON string allows users to specify aspects of a crawler's behavior. For more information, see Setting crawler configuration options.
  - name: CrawlerSecurityConfiguration
    value: "{{ CrawlerSecurityConfiguration }}"
    description: |
      The name of the SecurityConfiguration structure to be used by this crawler.
  - name: Tags
    value: "{{ Tags }}"
    description: |
      The tags to use with this crawler request. You may use tags to limit access to the crawler. For more information about tags in Glue, see Amazon Web Services Tags in Glue in the developer guide.

UPDATE examples

update_crawler_schedule

Updates the schedule of a crawler using a cron expression.

UPDATE aws.glue.crawlers
SET 
CrawlerName = '{{ CrawlerName }}',
Schedule = '{{ Schedule }}'
WHERE 
region = '{{ region }}' --required
AND CrawlerName = '{{ CrawlerName }}' --required;

update_crawler

Updates a crawler. If a crawler is running, you must stop it using StopCrawler before updating it.

UPDATE aws.glue.crawlers
SET 
Name = '{{ Name }}',
Role = '{{ Role }}',
DatabaseName = '{{ DatabaseName }}',
Description = '{{ Description }}',
Targets = '{{ Targets }}',
Schedule = '{{ Schedule }}',
Classifiers = '{{ Classifiers }}',
TablePrefix = '{{ TablePrefix }}',
SchemaChangePolicy = '{{ SchemaChangePolicy }}',
RecrawlPolicy = '{{ RecrawlPolicy }}',
LineageConfiguration = '{{ LineageConfiguration }}',
LakeFormationConfiguration = '{{ LakeFormationConfiguration }}',
Configuration = '{{ Configuration }}',
CrawlerSecurityConfiguration = '{{ CrawlerSecurityConfiguration }}'
WHERE 
region = '{{ region }}' --required;

DELETE examples

delete_crawler

Removes a specified crawler from the Glue Data Catalog, unless the crawler state is RUNNING.

DELETE FROM aws.glue.crawlers
WHERE region = '{{ region }}' --required
;

Lifecycle Methods

get_crawlers

Retrieves metadata for all crawlers defined in the customer account.

EXEC aws.glue.crawlers.get_crawlers 
@region='{{ region }}' --required 
@@json=
'{
"MaxResults": {{ MaxResults }}, 
"NextToken": "{{ NextToken }}"
}'
;

list_crawlers

Retrieves the names of all crawler resources in this Amazon Web Services account, or the resources with the specified tag. This operation allows you to see which resources are available in your account, and their names. This operation takes the optional Tags field, which you can use as a filter on the response so that tagged resources can be retrieved as a group. If you choose to use tags filtering, only resources with the tag are retrieved.

EXEC aws.glue.crawlers.list_crawlers 
@region='{{ region }}' --required 
@@json=
'{
"MaxResults": {{ MaxResults }}, 
"NextToken": "{{ NextToken }}", 
"Tags": "{{ Tags }}"
}'
;

start_crawler_schedule

Changes the schedule state of the specified crawler to SCHEDULED, unless the crawler is already running or the schedule state is already SCHEDULED.

EXEC aws.glue.crawlers.start_crawler_schedule 
@region='{{ region }}' --required 
@@json=
'{
"CrawlerName": "{{ CrawlerName }}"
}'
;

stop_crawler_schedule

Sets the schedule state of the specified crawler to NOT_SCHEDULED, but does not stop the crawler if it is already running.

EXEC aws.glue.crawlers.stop_crawler_schedule 
@region='{{ region }}' --required 
@@json=
'{
"CrawlerName": "{{ CrawlerName }}"
}'
;