datasets

Creates, updates, deletes, gets or lists a datasets resource.

Overview

Name	datasets
Type	Resource
Id	aws.lookoutequipment.datasets

Fields

The following fields are returned by SELECT queries:

describe_dataset

Name	Datatype	Description
created_at	string (date-time)	Specifies the time the dataset was created in Lookout for Equipment.
data_end_time	string (date-time)	Indicates the latest timestamp corresponding to data that was successfully ingested during the most recent ingestion of this particular dataset.
data_quality_summary	object	Gives statistics associated with the given dataset for the latest successful associated ingestion job id. These statistics primarily relate to quantifying incorrect data such as MissingCompleteSensorData, MissingSensorData, UnsupportedDateFormats, InsufficientSensorData, and DuplicateTimeStamps.
data_start_time	string (date-time)	Indicates the earliest timestamp corresponding to data that was successfully ingested during the most recent ingestion of this particular dataset.
dataset_arn	string	The Amazon Resource Name (ARN) of the dataset being described. (pattern: <code>arn:aws(-[^:]+)?:lookoutequipment:[a-zA-Z0-9-]*:[0-9]{12}:dataset/[0-9a-zA-Z_-]{1,200}/.+</code>)
dataset_name	string	The name of the dataset being described. (pattern: <code>^[0-9a-zA-Z_-]{1,200}$</code>)
ingested_files_summary	object	Gives statistics about how many files have been ingested, and which files have not been ingested, for a particular ingestion job.
ingestion_input_configuration	object	Specifies the S3 location configuration for the data input for the data ingestion job.
last_updated_at	string (date-time)	Specifies the time the dataset was last updated, if it was.
role_arn	string	The Amazon Resource Name (ARN) of the IAM role that you are using for this the data ingestion job. (pattern: <code>arn:aws(-[^:]+)?:iam::[0-9]{12}:role/.+</code>)
schema	string	A JSON description of the data that is in each time series dataset, including names, column names, and data types.
server_side_kms_key_id	string	Provides the identifier of the KMS key used to encrypt dataset data by Amazon Lookout for Equipment. (pattern: <code>arn:aws[a-z-]:kms:[a-z0-9-]:\d{12}:[\w-/]+</code>)
source_dataset_arn	string	The Amazon Resource Name (ARN) of the source dataset from which the current data being described was imported from. (pattern: <code>arn:aws(-[^:]+)?:lookoutequipment:[a-zA-Z0-9-]*:[0-9]{12}:dataset/[0-9a-zA-Z_-]{1,200}/.+</code>)
status	string	Indicates the status of the dataset. (CREATED, INGESTION_IN_PROGRESS, ACTIVE, IMPORT_IN_PROGRESS)

list_datasets

Name	Datatype	Description
dataset_summaries	array	Provides information about the specified dataset, including creation time, dataset ARN, and status.
next_token	string	An opaque pagination token indicating where to continue the listing of datasets. (pattern: <code>\p{ASCII}{0,8192}</code>)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
describe_dataset	select	region		Provides a JSON description of the data in each time series dataset, including names, column names, and data types.
list_datasets	select	region		Lists all datasets currently available in your account, filtering on the dataset name.
create_dataset	insert	region, DatasetName, ClientToken		Creates a container for a collection of data being ingested for analysis. The dataset contains the metadata describing where the data is and what the data actually looks like. For example, it contains the location of the data source, the data schema, and other information. A dataset also contains any tags associated with the ingested data.
delete_dataset	delete	region		Deletes a dataset and associated artifacts. The operation will check to see if any inference scheduler or data ingestion job is currently using the dataset, and if there isn't, the dataset, its metadata, and any associated data stored in S3 will be deleted. This does not affect any models that used this dataset for training and evaluation, but does prevent it from being used in the future.
start_data_ingestion_job	exec	region, DatasetName, IngestionInputConfiguration, RoleArn, ClientToken		Starts a data ingestion job. Amazon Lookout for Equipment returns the job status.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
region	string	AWS region (default: us-east-1)

SELECT examples

describe_dataset

Provides a JSON description of the data in each time series dataset, including names, column names, and data types.

SELECT
created_at,
data_end_time,
data_quality_summary,
data_start_time,
dataset_arn,
dataset_name,
ingested_files_summary,
ingestion_input_configuration,
last_updated_at,
role_arn,
schema,
server_side_kms_key_id,
source_dataset_arn,
status
FROM aws.lookoutequipment.datasets
WHERE region = '{{ region }}' -- required
;

list_datasets

Lists all datasets currently available in your account, filtering on the dataset name.

SELECT
dataset_summaries,
next_token
FROM aws.lookoutequipment.datasets
WHERE region = '{{ region }}' -- required
;

INSERT examples

create_dataset

Creates a container for a collection of data being ingested for analysis. The dataset contains the metadata describing where the data is and what the data actually looks like. For example, it contains the location of the data source, the data schema, and other information. A dataset also contains any tags associated with the ingested data.

INSERT INTO aws.lookoutequipment.datasets (
DatasetName,
DatasetSchema,
ServerSideKmsKeyId,
ClientToken,
Tags,
region
)
SELECT 
'{{ DatasetName }}' /* required */,
'{{ DatasetSchema }}',
'{{ ServerSideKmsKeyId }}',
'{{ ClientToken }}' /* required */,
'{{ Tags }}',
'{{ region }}'
RETURNING
dataset_arn,
dataset_name,
status
;

Manifest

# Description fields are for documentation purposes
- name: datasets
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the datasets resource.
  - name: DatasetName
    value: "{{ DatasetName }}"
    description: |
      The name of the dataset being created.
  - name: DatasetSchema
    description: |
      A JSON description of the data that is in each time series dataset, including names, column names, and data types.
    value:
      InlineDataSchema: "{{ InlineDataSchema }}"
  - name: ServerSideKmsKeyId
    value: "{{ ServerSideKmsKeyId }}"
    description: |
      Provides the identifier of the KMS key used to encrypt dataset data by Amazon Lookout for Equipment.
  - name: ClientToken
    value: "{{ ClientToken }}"
    description: |
      A unique identifier for the request. If you do not set the client request token, Amazon Lookout for Equipment generates one.
  - name: Tags
    description: |
      Any tags associated with the ingested data described in the dataset.
    value:
      - Key: "{{ Key }}"
        Value: "{{ Value }}"

DELETE examples

delete_dataset

Deletes a dataset and associated artifacts. The operation will check to see if any inference scheduler or data ingestion job is currently using the dataset, and if there isn't, the dataset, its metadata, and any associated data stored in S3 will be deleted. This does not affect any models that used this dataset for training and evaluation, but does prevent it from being used in the future.

DELETE FROM aws.lookoutequipment.datasets
WHERE region = '{{ region }}' --required
;

Lifecycle Methods

start_data_ingestion_job

Starts a data ingestion job. Amazon Lookout for Equipment returns the job status.

EXEC aws.lookoutequipment.datasets.start_data_ingestion_job 
@region='{{ region }}' --required 
@@json=
'{
"DatasetName": "{{ DatasetName }}", 
"IngestionInputConfiguration": "{{ IngestionInputConfiguration }}", 
"RoleArn": "{{ RoleArn }}", 
"ClientToken": "{{ ClientToken }}"
}'
;