Skip to main content

data_sources

Creates, updates, deletes, gets or lists a data_sources resource.

Overview​

Namedata_sources
TypeResource
Idaws.machinelearning.data_sources

Fields​

The following fields are returned by SELECT queries:

NameDatatypeDescription
compute_statisticsbooleanThe parameter is true if statistics need to be generated from the observation data.
compute_timeinteger (int64)Long integer type that is a 64-bit signed number.
created_atstring (date-time)A timestamp represented in epoch time.
created_by_iam_userstringThe AWS user account from which the DataSource was created. The account type can be either an AWS root account or an AWS Identity and Access Management (IAM) user account. (pattern: <code>arn:aws:iam::[0-9]+:((user/.+)|(root))</code>)
data_location_s3stringThe location of the data file or directory in Amazon Simple Storage Service (Amazon S3). (pattern: <code>s3:​//([^/]+)(/.*)?</code>)
data_rearrangementstringA JSON string that represents the splitting and rearrangement requirement used when this DataSource was created.
data_size_in_bytesinteger (int64)Long integer type that is a 64-bit signed number.
data_source_idstringThe ID assigned to the DataSource at creation. This value should be identical to the value of the DataSourceId in the request. (pattern: <code>[a-zA-Z0-9_.-]+</code>)
data_source_schemastringThe schema used by all of the data files of this DataSource. Note: This parameter is provided as part of the verbose format.
finished_atstring (date-time)A timestamp represented in epoch time.
last_updated_atstring (date-time)A timestamp represented in epoch time.
log_uristringA link to the file containing logs of CreateDataSourceFrom* operations.
messagestringThe user-supplied description of the most recent details about creating the DataSource.
namestringA user-supplied name or description of the DataSource. (pattern: <code>.\S.|^$</code>)
number_of_filesinteger (int64)Long integer type that is a 64-bit signed number.
rds_metadataobjectThe datasource details that are specific to Amazon RDS.
redshift_metadataobjectDescribes the DataSource details specific to Amazon Redshift.
role_arnstringThe Amazon Resource Name (ARN) of an AWS IAM Role, such as the following: arn:aws:iam::account:role/rolename.
started_atstring (date-time)A timestamp represented in epoch time.
statusstringThe current status of the DataSource. This element can have one of the following values: PENDING - Amazon ML submitted a request to create a DataSource. INPROGRESS - The creation process is underway. FAILED - The request to create a DataSource did not run to completion. It is not usable. COMPLETED - The creation process completed successfully. DELETED - The DataSource is marked as deleted. It is not usable. (PENDING, INPROGRESS, FAILED, COMPLETED, DELETED)

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
get_data_sourceselectregionReturns a DataSource that includes metadata and data file information, as well as the current status of the DataSource. GetDataSource provides results in normal or verbose format. The verbose format adds the schema description and the list of files pointed to by the DataSource to the normal format.
describe_data_sourcesselectregionReturns a list of DataSource that match the search criteria in the request.
create_data_source_from_rdsinsertregion, DataSourceId, RDSData, RoleARNCreates a DataSource object from an Amazon Relational Database Service (Amazon RDS). A DataSource references data that can be used to perform CreateMLModel, CreateEvaluation, or CreateBatchPrediction operations. CreateDataSourceFromRDS is an asynchronous operation. In response to CreateDataSourceFromRDS, Amazon Machine Learning (Amazon ML) immediately returns and sets the DataSource status to PENDING. After the DataSource is created and ready for use, Amazon ML sets the Status parameter to COMPLETED. DataSource in the COMPLETED or PENDING state can be used only to perform >CreateMLModel>, CreateEvaluation, or CreateBatchPrediction operations. If Amazon ML cannot accept the input source, it sets the Status parameter to FAILED and includes an error message in the Message attribute of the GetDataSource operation response.
create_data_source_from_redshiftinsertregion, DataSourceId, DataSpec, RoleARNCreates a DataSource from a database hosted on an Amazon Redshift cluster. A DataSource references data that can be used to perform either CreateMLModel, CreateEvaluation, or CreateBatchPrediction operations. CreateDataSourceFromRedshift is an asynchronous operation. In response to CreateDataSourceFromRedshift, Amazon Machine Learning (Amazon ML) immediately returns and sets the DataSource status to PENDING. After the DataSource is created and ready for use, Amazon ML sets the Status parameter to COMPLETED. DataSource in COMPLETED or PENDING states can be used to perform only CreateMLModel, CreateEvaluation, or CreateBatchPrediction operations. If Amazon ML can't accept the input source, it sets the Status parameter to FAILED and includes an error message in the Message attribute of the GetDataSource operation response. The observations should be contained in the database hosted on an Amazon Redshift cluster and should be specified by a SelectSqlQuery query. Amazon ML executes an Unload command in Amazon Redshift to transfer the result set of the SelectSqlQuery query to S3StagingLocation. After the DataSource has been created, it's ready for use in evaluations and batch predictions. If you plan to use the DataSource to train an MLModel, the DataSource also requires a recipe. A recipe describes how each input variable will be used in training an MLModel. Will the variable be included or excluded from training? Will the variable be manipulated; for example, will it be combined with another variable or will it be split apart into word combinations? The recipe provides answers to these questions. You can't change an existing datasource, but you can copy and modify the settings from an existing Amazon Redshift datasource to create a new datasource. To do so, call GetDataSource for an existing datasource and copy the values to a CreateDataSource call. Change the settings that you want to change and make sure that all required fields have the appropriate values.
create_data_source_from_s3insertregion, DataSourceId, DataSpecCreates a DataSource object. A DataSource references data that can be used to perform CreateMLModel, CreateEvaluation, or CreateBatchPrediction operations. CreateDataSourceFromS3 is an asynchronous operation. In response to CreateDataSourceFromS3, Amazon Machine Learning (Amazon ML) immediately returns and sets the DataSource status to PENDING. After the DataSource has been created and is ready for use, Amazon ML sets the Status parameter to COMPLETED. DataSource in the COMPLETED or PENDING state can be used to perform only CreateMLModel, CreateEvaluation or CreateBatchPrediction operations. If Amazon ML can't accept the input source, it sets the Status parameter to FAILED and includes an error message in the Message attribute of the GetDataSource operation response. The observation data used in a DataSource should be ready to use; that is, it should have a consistent structure, and missing data values should be kept to a minimum. The observation data must reside in one or more .csv files in an Amazon Simple Storage Service (Amazon S3) location, along with a schema that describes the data items by name and type. The same schema must be used for all of the data files referenced by the DataSource. After the DataSource has been created, it's ready to use in evaluations and batch predictions. If you plan to use the DataSource to train an MLModel, the DataSource also needs a recipe. A recipe describes how each input variable will be used in training an MLModel. Will the variable be included or excluded from training? Will the variable be manipulated; for example, will it be combined with another variable or will it be split apart into word combinations? The recipe provides answers to these questions.
update_data_sourceupdateregion, DataSourceId, DataSourceNameUpdates the DataSourceName of a DataSource. You can use the GetDataSource operation to view the contents of the updated data element.
delete_data_sourcedeleteregionAssigns the DELETED status to a DataSource, rendering it unusable. After using the DeleteDataSource operation, you can use the GetDataSource operation to verify that the status of the DataSource changed to DELETED. Caution: The results of the DeleteDataSource operation are irreversible.

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
regionstringAWS region (default: us-east-1)

SELECT examples​

Returns a DataSource that includes metadata and data file information, as well as the current status of the DataSource. GetDataSource provides results in normal or verbose format. The verbose format adds the schema description and the list of files pointed to by the DataSource to the normal format.

SELECT
compute_statistics,
compute_time,
created_at,
created_by_iam_user,
data_location_s3,
data_rearrangement,
data_size_in_bytes,
data_source_id,
data_source_schema,
finished_at,
last_updated_at,
log_uri,
message,
name,
number_of_files,
rds_metadata,
redshift_metadata,
role_arn,
started_at,
status
FROM aws.machinelearning.data_sources
WHERE region = '{{ region }}' -- required
;

INSERT examples​

Creates a DataSource object from an Amazon Relational Database Service (Amazon RDS). A DataSource references data that can be used to perform CreateMLModel, CreateEvaluation, or CreateBatchPrediction operations. CreateDataSourceFromRDS is an asynchronous operation. In response to CreateDataSourceFromRDS, Amazon Machine Learning (Amazon ML) immediately returns and sets the DataSource status to PENDING. After the DataSource is created and ready for use, Amazon ML sets the Status parameter to COMPLETED. DataSource in the COMPLETED or PENDING state can be used only to perform >CreateMLModel>, CreateEvaluation, or CreateBatchPrediction operations. If Amazon ML cannot accept the input source, it sets the Status parameter to FAILED and includes an error message in the Message attribute of the GetDataSource operation response.

INSERT INTO aws.machinelearning.data_sources (
DataSourceId,
DataSourceName,
RDSData,
RoleARN,
ComputeStatistics,
region
)
SELECT 
'{{ DataSourceId }}' /* required */,
'{{ DataSourceName }}',
'{{ RDSData }}' /* required */,
'{{ RoleARN }}' /* required */,
{{ ComputeStatistics }},
'{{ region }}'
RETURNING
data_source_id
;

UPDATE examples​

Updates the DataSourceName of a DataSource. You can use the GetDataSource operation to view the contents of the updated data element.

UPDATE aws.machinelearning.data_sources
SET 
DataSourceId = '{{ DataSourceId }}',
DataSourceName = '{{ DataSourceName }}'
WHERE 
region = '{{ region }}' --required
AND DataSourceId = '{{ DataSourceId }}' --required
AND DataSourceName = '{{ DataSourceName }}' --required
RETURNING
data_source_id;

DELETE examples​

Assigns the DELETED status to a DataSource, rendering it unusable. After using the DeleteDataSource operation, you can use the GetDataSource operation to verify that the status of the DataSource changed to DELETED. Caution: The results of the DeleteDataSource operation are irreversible.

DELETE FROM aws.machinelearning.data_sources
WHERE region = '{{ region }}' --required
;
Last updated on