datasets

Creates, updates, deletes, gets or lists a datasets resource.

Overview

Name	datasets
Type	Resource
Id	aws.databrew.datasets

Fields

The following fields are returned by SELECT queries:

describe_dataset

Name	Datatype	Description
create_date	string (date-time)	The date and time that the dataset was created.
created_by	string	The identifier (user name) of the user who created the dataset.
format	string	The file format of a dataset that is created from an Amazon S3 file or folder. (CSV, JSON, PARQUET, EXCEL, ORC)
format_options	object	Represents a set of options that define the structure of either comma-separated value (CSV), Excel, or JSON input.
input	object	Represents information on how DataBrew can find data, in either the Glue Data Catalog or Amazon S3.
last_modified_by	string	The identifier (user name) of the user who last modified the dataset.
last_modified_date	string (date-time)	The date and time that the dataset was last modified.
name	string	The name of the dataset.
path_options	object	Represents a set of options that define how DataBrew selects files for a given Amazon S3 path in a dataset.
resource_arn	string	The Amazon Resource Name (ARN) of the dataset.
source	string	The location of the data for this dataset, Amazon S3 or the Glue Data Catalog. (S3, DATA-CATALOG, DATABASE)
tags	object	Metadata tags associated with this dataset.

list_datasets

Name	Datatype	Description
account_id	string	The ID of the Amazon Web Services account that owns the dataset.
create_date	string (date-time)	The date and time that the dataset was created.
created_by	string	The Amazon Resource Name (ARN) of the user who created the dataset.
format	string	The file format of a dataset that is created from an Amazon S3 file or folder. (CSV, JSON, PARQUET, EXCEL, ORC)
format_options	object	Represents a set of options that define the structure of either comma-separated value (CSV), Excel, or JSON input.
input	object	Represents information on how DataBrew can find data, in either the Glue Data Catalog or Amazon S3.
last_modified_by	string	The Amazon Resource Name (ARN) of the user who last modified the dataset.
last_modified_date	string (date-time)	The last modification date and time of the dataset.
name	string	The unique name of the dataset.
path_options	object	Represents a set of options that define how DataBrew selects files for a given Amazon S3 path in a dataset.
resource_arn	string	The unique Amazon Resource Name (ARN) for the dataset.
source	string	The location of the data for the dataset, either Amazon S3 or the Glue Data Catalog. (S3, DATA-CATALOG, DATABASE)
tags	object	Metadata tags that have been applied to the dataset.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
describe_dataset	select	name, region		Returns the definition of a specific DataBrew dataset.
list_datasets	select	region	maxResults, nextToken	Lists all of the DataBrew datasets.
create_dataset	insert	region		Creates a new DataBrew dataset.
update_dataset	update	name, region		Modifies the definition of an existing DataBrew dataset.
delete_dataset	delete	name, region		Deletes a dataset from DataBrew.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
name	string	The name of the dataset to be deleted.
region	string	AWS region (default: us-east-1)
maxResults	integer	The maximum number of results to return in this request.
nextToken	string	The token returned by a previous call to retrieve the next set of results.

SELECT examples

describe_dataset

Returns the definition of a specific DataBrew dataset.

SELECT
create_date,
created_by,
format,
format_options,
input,
last_modified_by,
last_modified_date,
name,
path_options,
resource_arn,
source,
tags
FROM aws.databrew.datasets
WHERE name = '{{ name }}' -- required
AND region = '{{ region }}' -- required
;

list_datasets

Lists all of the DataBrew datasets.

SELECT
account_id,
create_date,
created_by,
format,
format_options,
input,
last_modified_by,
last_modified_date,
name,
path_options,
resource_arn,
source,
tags
FROM aws.databrew.datasets
WHERE region = '{{ region }}' -- required
AND maxResults = '{{ maxResults }}'
AND nextToken = '{{ nextToken }}'
;

INSERT examples

create_dataset

Creates a new DataBrew dataset.

INSERT INTO aws.databrew.datasets (
Name,
Format,
FormatOptions,
Input,
PathOptions,
Tags,
region
)
SELECT 
'{{ Name }}',
'{{ Format }}',
'{{ FormatOptions }}',
'{{ Input }}',
'{{ PathOptions }}',
'{{ Tags }}',
'{{ region }}'
RETURNING
name
;

Manifest

# Description fields are for documentation purposes
- name: datasets
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the datasets resource.
  - name: Name
    value: "{{ Name }}"
  - name: Format
    value: "{{ Format }}"
    valid_values: ['CSV', 'JSON', 'PARQUET', 'EXCEL', 'ORC']
  - name: FormatOptions
    description: |
      Represents a set of options that define the structure of either comma-separated value (CSV), Excel, or JSON input.
    value:
      Json:
        MultiLine: {{ MultiLine }}
      Excel:
        SheetNames:
          - "{{ SheetNames }}"
        SheetIndexes:
          - {{ SheetIndexes }}
        HeaderRow: {{ HeaderRow }}
      Csv:
        Delimiter: "{{ Delimiter }}"
        HeaderRow: {{ HeaderRow }}
  - name: Input
    description: |
      Represents information on how DataBrew can find data, in either the Glue Data Catalog or Amazon S3.
    value:
      S3InputDefinition:
        Bucket: "{{ Bucket }}"
        Key: "{{ Key }}"
        BucketOwner: "{{ BucketOwner }}"
      DataCatalogInputDefinition:
        CatalogId: "{{ CatalogId }}"
        DatabaseName: "{{ DatabaseName }}"
        TableName: "{{ TableName }}"
        TempDirectory:
          Bucket: "{{ Bucket }}"
          Key: "{{ Key }}"
          BucketOwner: "{{ BucketOwner }}"
      DatabaseInputDefinition:
        GlueConnectionName: "{{ GlueConnectionName }}"
        DatabaseTableName: "{{ DatabaseTableName }}"
        TempDirectory:
          Bucket: "{{ Bucket }}"
          Key: "{{ Key }}"
          BucketOwner: "{{ BucketOwner }}"
        QueryString: "{{ QueryString }}"
      Metadata:
        SourceArn: "{{ SourceArn }}"
  - name: PathOptions
    description: |
      Represents a set of options that define how DataBrew selects files for a given Amazon S3 path in a dataset.
    value:
      LastModifiedDateCondition:
        Expression: "{{ Expression }}"
        ValuesMap: "{{ ValuesMap }}"
      FilesLimit:
        MaxFiles: {{ MaxFiles }}
        OrderedBy: "{{ OrderedBy }}"
        Order: "{{ Order }}"
      Parameters: "{{ Parameters }}"
  - name: Tags
    value: "{{ Tags }}"

UPDATE examples

update_dataset

Modifies the definition of an existing DataBrew dataset.

UPDATE aws.databrew.datasets
SET 
Format = '{{ Format }}',
FormatOptions = '{{ FormatOptions }}',
Input = '{{ Input }}',
PathOptions = '{{ PathOptions }}'
WHERE 
name = '{{ name }}' --required
AND region = '{{ region }}' --required
RETURNING
name;

DELETE examples

delete_dataset

Deletes a dataset from DataBrew.

DELETE FROM aws.databrew.datasets
WHERE name = '{{ name }}' --required
AND region = '{{ region }}' --required
;