jobs

Creates, updates, deletes, gets or lists a jobs resource.

Overview

Name	jobs
Type	Resource
Id	aws.databrew.jobs

Fields

The following fields are returned by SELECT queries:

describe_job

Name	Datatype	Description
create_date	string (date-time)	The date and time that the job was created.
created_by	string	The identifier (user name) of the user associated with the creation of the job.
data_catalog_outputs	array	One or more artifacts that represent the Glue Data Catalog output from running the job.
database_outputs	array	Represents a list of JDBC database output objects which defines the output destination for a DataBrew recipe job to write into.
dataset_name	string	The dataset that the job acts upon.
encryption_key_arn	string	The Amazon Resource Name (ARN) of an encryption key that is used to protect the job.
encryption_mode	string	The encryption mode for the job, which can be one of the following: SSE-KMS - Server-side encryption with keys managed by KMS. SSE-S3 - Server-side encryption with keys managed by Amazon S3. (SSE-KMS, SSE-S3)
job_sample	object	A sample configuration for profile jobs only, which determines the number of rows on which the profile job is run. If a JobSample value isn't provided, the default is used. The default value is CUSTOM_ROWS for the mode parameter and 20,000 for the size parameter.
last_modified_by	string	The identifier (user name) of the user who last modified the job.
last_modified_date	string (date-time)	The date and time that the job was last modified.
log_subscription	string	Indicates whether Amazon CloudWatch logging is enabled for this job. (ENABLE, DISABLE)
max_capacity	integer	The maximum number of compute nodes that DataBrew can consume when the job processes data.
max_retries	integer	The maximum number of times to retry the job after a job run fails.
name	string	The name of the job.
outputs	array	One or more artifacts that represent the output from running the job.
profile_configuration	object	Configuration for profile jobs. Configuration can be used to select columns, do evaluations, and override default parameters of evaluations. When configuration is undefined, the profile job will apply default settings to all supported columns.
project_name	string	The DataBrew project associated with this job.
recipe_reference	object	Represents the name and version of a DataBrew recipe.
resource_arn	string	The Amazon Resource Name (ARN) of the job.
role_arn	string	The ARN of the Identity and Access Management (IAM) role to be assumed when DataBrew runs the job.
tags	object	Metadata tags associated with this job.
timeout	integer	The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT.
type	string	The job type, which must be one of the following: PROFILE - The job analyzes the dataset to determine its size, data types, data distribution, and more. RECIPE - The job applies one or more transformations to a dataset. (PROFILE, RECIPE)
validation_configurations	array	List of validation configurations that are applied to the profile job.

list_jobs

Name	Datatype	Description
account_id	string	The ID of the Amazon Web Services account that owns the job.
create_date	string (date-time)	The date and time that the job was created.
created_by	string	The Amazon Resource Name (ARN) of the user who created the job.
data_catalog_outputs	array	One or more artifacts that represent the Glue Data Catalog output from running the job.
database_outputs	array	Represents a list of JDBC database output objects which defines the output destination for a DataBrew recipe job to write into.
dataset_name	string	A dataset that the job is to process.
encryption_key_arn	string	The Amazon Resource Name (ARN) of an encryption key that is used to protect the job output. For more information, see Encrypting data written by DataBrew jobs
encryption_mode	string	The encryption mode for the job, which can be one of the following: SSE-KMS - Server-side encryption with keys managed by KMS. SSE-S3 - Server-side encryption with keys managed by Amazon S3. (SSE-KMS, SSE-S3)
job_sample	object	A sample configuration for profile jobs only, which determines the number of rows on which the profile job is run. If a JobSample value isn't provided, the default is used. The default value is CUSTOM_ROWS for the mode parameter and 20,000 for the size parameter.
last_modified_by	string	The Amazon Resource Name (ARN) of the user who last modified the job.
last_modified_date	string (date-time)	The modification date and time of the job.
log_subscription	string	The current status of Amazon CloudWatch logging for the job. (ENABLE, DISABLE)
max_capacity	integer	The maximum number of nodes that can be consumed when the job processes data.
max_retries	integer	The maximum number of times to retry the job after a job run fails.
name	string	The unique name of the job.
outputs	array	One or more artifacts that represent output from running the job.
project_name	string	The name of the project that the job is associated with.
recipe_reference	object	Represents the name and version of a DataBrew recipe.
resource_arn	string	The unique Amazon Resource Name (ARN) for the job.
role_arn	string	The Amazon Resource Name (ARN) of the role to be assumed for this job.
tags	object	Metadata tags that have been applied to the job.
timeout	integer	The job's timeout in minutes. A job that attempts to run longer than this timeout period ends with a status of TIMEOUT.
type	string	The job type of the job, which must be one of the following: PROFILE - A job to analyze a dataset, to determine its size, data types, data distribution, and more. RECIPE - A job to apply one or more transformations to a dataset. (PROFILE, RECIPE)
validation_configurations	array	List of validation configurations that are applied to the profile job.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
describe_job	select	name, region		Returns the definition of a specific DataBrew job.
list_jobs	select	region	datasetName, maxResults, nextToken, projectName	Lists all of the DataBrew jobs that are defined.
create_profile_job	insert	region, DatasetName, OutputLocation, RoleArn		Creates a new job to analyze a dataset and create its data profile.
create_recipe_job	insert	region, RoleArn		Creates a new job to transform input data, using steps defined in an existing Glue DataBrew recipe
update_profile_job	update	name, region, OutputLocation, RoleArn		Modifies the definition of an existing profile job.
update_recipe_job	update	name, region, RoleArn		Modifies the definition of an existing DataBrew recipe job.
delete_job	delete	name, region		Deletes the specified DataBrew job.
start_job_run	exec	name, region		Runs a DataBrew job.
stop_job_run	exec	name, run_id, region		Stops a particular run of a job.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
name	string	The name of the job to be stopped.
region	string	AWS region (default: us-east-1)
run_id	string	The ID of the job run to be stopped.
datasetName	string	The name of a dataset. Using this parameter indicates to return only those jobs that act on the specified dataset.
maxResults	integer	The maximum number of results to return in this request.
nextToken	string	A token generated by DataBrew that specifies where to continue pagination if a previous request was truncated. To get the next set of pages, pass in the NextToken value from the response object of the previous page call.
projectName	string	The name of a project. Using this parameter indicates to return only those jobs that are associated with the specified project.

SELECT examples

describe_job

Returns the definition of a specific DataBrew job.

SELECT
create_date,
created_by,
data_catalog_outputs,
database_outputs,
dataset_name,
encryption_key_arn,
encryption_mode,
job_sample,
last_modified_by,
last_modified_date,
log_subscription,
max_capacity,
max_retries,
name,
outputs,
profile_configuration,
project_name,
recipe_reference,
resource_arn,
role_arn,
tags,
timeout,
type,
validation_configurations
FROM aws.databrew.jobs
WHERE name = '{{ name }}' -- required
AND region = '{{ region }}' -- required
;

list_jobs

Lists all of the DataBrew jobs that are defined.

SELECT
account_id,
create_date,
created_by,
data_catalog_outputs,
database_outputs,
dataset_name,
encryption_key_arn,
encryption_mode,
job_sample,
last_modified_by,
last_modified_date,
log_subscription,
max_capacity,
max_retries,
name,
outputs,
project_name,
recipe_reference,
resource_arn,
role_arn,
tags,
timeout,
type,
validation_configurations
FROM aws.databrew.jobs
WHERE region = '{{ region }}' -- required
AND datasetName = '{{ datasetName }}'
AND maxResults = '{{ maxResults }}'
AND nextToken = '{{ nextToken }}'
AND projectName = '{{ projectName }}'
;

INSERT examples

create_profile_job

Creates a new job to analyze a dataset and create its data profile.

INSERT INTO aws.databrew.jobs (
DatasetName,
EncryptionKeyArn,
EncryptionMode,
Name,
LogSubscription,
MaxCapacity,
MaxRetries,
OutputLocation,
Configuration,
ValidationConfigurations,
RoleArn,
Tags,
Timeout,
JobSample,
region
)
SELECT 
'{{ DatasetName }}' /* required */,
'{{ EncryptionKeyArn }}',
'{{ EncryptionMode }}',
'{{ Name }}',
'{{ LogSubscription }}',
{{ MaxCapacity }},
{{ MaxRetries }},
'{{ OutputLocation }}' /* required */,
'{{ Configuration }}',
'{{ ValidationConfigurations }}',
'{{ RoleArn }}' /* required */,
'{{ Tags }}',
{{ Timeout }},
'{{ JobSample }}',
'{{ region }}'
RETURNING
name
;

create_recipe_job

Creates a new job to transform input data, using steps defined in an existing Glue DataBrew recipe

INSERT INTO aws.databrew.jobs (
DatasetName,
EncryptionKeyArn,
EncryptionMode,
Name,
LogSubscription,
MaxCapacity,
MaxRetries,
Outputs,
DataCatalogOutputs,
DatabaseOutputs,
ProjectName,
RecipeReference,
RoleArn,
Tags,
Timeout,
region
)
SELECT 
'{{ DatasetName }}',
'{{ EncryptionKeyArn }}',
'{{ EncryptionMode }}',
'{{ Name }}',
'{{ LogSubscription }}',
{{ MaxCapacity }},
{{ MaxRetries }},
'{{ Outputs }}',
'{{ DataCatalogOutputs }}',
'{{ DatabaseOutputs }}',
'{{ ProjectName }}',
'{{ RecipeReference }}',
'{{ RoleArn }}' /* required */,
'{{ Tags }}',
{{ Timeout }},
'{{ region }}'
RETURNING
name
;

Manifest

# Description fields are for documentation purposes
- name: jobs
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the jobs resource.
  - name: DatasetName
    value: "{{ DatasetName }}"
  - name: EncryptionKeyArn
    value: "{{ EncryptionKeyArn }}"
  - name: EncryptionMode
    value: "{{ EncryptionMode }}"
    valid_values: ['SSE-KMS', 'SSE-S3']
  - name: Name
    value: "{{ Name }}"
  - name: LogSubscription
    value: "{{ LogSubscription }}"
    valid_values: ['ENABLE', 'DISABLE']
  - name: MaxCapacity
    value: {{ MaxCapacity }}
  - name: MaxRetries
    value: {{ MaxRetries }}
  - name: OutputLocation
    description: |
      Represents an Amazon S3 location (bucket name, bucket owner, and object key) where DataBrew can read input data, or write output from a job.
    value:
      Bucket: "{{ Bucket }}"
      Key: "{{ Key }}"
      BucketOwner: "{{ BucketOwner }}"
  - name: Configuration
    description: |
      Configuration for profile jobs. Configuration can be used to select columns, do evaluations, and override default parameters of evaluations. When configuration is undefined, the profile job will apply default settings to all supported columns.
    value:
      DatasetStatisticsConfiguration:
        IncludedStatistics:
          - "{{ IncludedStatistics }}"
        Overrides:
          - Statistic: "{{ Statistic }}"
            Parameters: "{{ Parameters }}"
      ProfileColumns:
        - Regex: "{{ Regex }}"
          Name: "{{ Name }}"
      ColumnStatisticsConfigurations:
        - Selectors: "{{ Selectors }}"
          Statistics:
            IncludedStatistics:
              - "{{ IncludedStatistics }}"
            Overrides:
              - Statistic: "{{ Statistic }}"
                Parameters: "{{ Parameters }}"
      EntityDetectorConfiguration:
        EntityTypes:
          - "{{ EntityTypes }}"
        AllowedStatistics:
          - Statistics: "{{ Statistics }}"
  - name: ValidationConfigurations
    value:
      - RulesetArn: "{{ RulesetArn }}"
        ValidationMode: "{{ ValidationMode }}"
  - name: RoleArn
    value: "{{ RoleArn }}"
  - name: Tags
    value: "{{ Tags }}"
  - name: Timeout
    value: {{ Timeout }}
  - name: JobSample
    description: |
      A sample configuration for profile jobs only, which determines the number of rows on which the profile job is run. If a JobSample value isn't provided, the default is used. The default value is CUSTOM_ROWS for the mode parameter and 20,000 for the size parameter.
    value:
      Mode: "{{ Mode }}"
      Size: {{ Size }}
  - name: Outputs
    value:
      - CompressionFormat: "{{ CompressionFormat }}"
        Format: "{{ Format }}"
        PartitionColumns: "{{ PartitionColumns }}"
        Location:
          Bucket: "{{ Bucket }}"
          Key: "{{ Key }}"
          BucketOwner: "{{ BucketOwner }}"
        Overwrite: {{ Overwrite }}
        FormatOptions:
          Csv:
            Delimiter: "{{ Delimiter }}"
        MaxOutputFiles: {{ MaxOutputFiles }}
  - name: DataCatalogOutputs
    value:
      - CatalogId: "{{ CatalogId }}"
        DatabaseName: "{{ DatabaseName }}"
        TableName: "{{ TableName }}"
        S3Options:
          Location:
            Bucket: "{{ Bucket }}"
            Key: "{{ Key }}"
            BucketOwner: "{{ BucketOwner }}"
        DatabaseOptions:
          TempDirectory:
            Bucket: "{{ Bucket }}"
            Key: "{{ Key }}"
            BucketOwner: "{{ BucketOwner }}"
          TableName: "{{ TableName }}"
        Overwrite: {{ Overwrite }}
  - name: DatabaseOutputs
    value:
      - GlueConnectionName: "{{ GlueConnectionName }}"
        DatabaseOptions:
          TempDirectory:
            Bucket: "{{ Bucket }}"
            Key: "{{ Key }}"
            BucketOwner: "{{ BucketOwner }}"
          TableName: "{{ TableName }}"
        DatabaseOutputMode: "{{ DatabaseOutputMode }}"
  - name: ProjectName
    value: "{{ ProjectName }}"
  - name: RecipeReference
    description: |
      Represents the name and version of a DataBrew recipe.
    value:
      Name: "{{ Name }}"
      RecipeVersion: "{{ RecipeVersion }}"

UPDATE examples

update_profile_job

Modifies the definition of an existing profile job.

UPDATE aws.databrew.jobs
SET 
Configuration = '{{ Configuration }}',
EncryptionKeyArn = '{{ EncryptionKeyArn }}',
EncryptionMode = '{{ EncryptionMode }}',
LogSubscription = '{{ LogSubscription }}',
MaxCapacity = {{ MaxCapacity }},
MaxRetries = {{ MaxRetries }},
OutputLocation = '{{ OutputLocation }}',
ValidationConfigurations = '{{ ValidationConfigurations }}',
RoleArn = '{{ RoleArn }}',
Timeout = {{ Timeout }},
JobSample = '{{ JobSample }}'
WHERE 
name = '{{ name }}' --required
AND region = '{{ region }}' --required
AND OutputLocation = '{{ OutputLocation }}' --required
AND RoleArn = '{{ RoleArn }}' --required
RETURNING
name;

update_recipe_job

Modifies the definition of an existing DataBrew recipe job.

UPDATE aws.databrew.jobs
SET 
EncryptionKeyArn = '{{ EncryptionKeyArn }}',
EncryptionMode = '{{ EncryptionMode }}',
LogSubscription = '{{ LogSubscription }}',
MaxCapacity = {{ MaxCapacity }},
MaxRetries = {{ MaxRetries }},
Outputs = '{{ Outputs }}',
DataCatalogOutputs = '{{ DataCatalogOutputs }}',
DatabaseOutputs = '{{ DatabaseOutputs }}',
RoleArn = '{{ RoleArn }}',
Timeout = {{ Timeout }}
WHERE 
name = '{{ name }}' --required
AND region = '{{ region }}' --required
AND RoleArn = '{{ RoleArn }}' --required
RETURNING
name;

DELETE examples

delete_job

Deletes the specified DataBrew job.

DELETE FROM aws.databrew.jobs
WHERE name = '{{ name }}' --required
AND region = '{{ region }}' --required
;

Lifecycle Methods

start_job_run

Runs a DataBrew job.

EXEC aws.databrew.jobs.start_job_run 
@name='{{ name }}' --required, 
@region='{{ region }}' --required
;

stop_job_run

Stops a particular run of a job.

EXEC aws.databrew.jobs.stop_job_run 
@name='{{ name }}' --required, 
@run_id='{{ run_id }}' --required, 
@region='{{ region }}' --required
;