Skip to main content

location_hdfs

Creates, updates, deletes, gets or lists a location_hdfs resource.

Overview​

Namelocation_hdfs
TypeResource
Idaws.datasync.location_hdfs

Fields​

The following fields are returned by SELECT queries:

NameDatatypeDescription
agent_arnsarrayThe ARNs of the DataSync agents that can connect with your HDFS cluster.
authentication_typestringThe type of authentication used to determine the identity of the user. (SIMPLE, KERBEROS)
block_sizeintegerThe size of the data blocks to write into the HDFS cluster.
cmk_secret_configobjectDescribes configuration information for a DataSync-managed secret, such as a KerberosKeytab that DataSync uses to access a specific storage location, with a customer-managed KMS key.
creation_timestring (date-time)The time that the HDFS location was created.
custom_secret_configobjectDescribes configuration information for a customer-managed secret, such as a KerberosKeytab that DataSync uses to access a specific storage location, with a customer-managed Identity and Access Management (IAM) role that provides access to the secret.
kerberos_principalstringThe Kerberos principal with access to the files and folders on the HDFS cluster. This parameter is used if the AuthenticationType is defined as KERBEROS. (pattern: <code>^.+$</code>)
kms_key_provider_uristringThe URI of the HDFS cluster's Key Management Server (KMS). (pattern: <code>^kms://http[s]?@(([a-zA-Z0-9-][a-zA-Z0-9]).)([A-Za-z0-9-][A-Za-z0-9])(;(([a-zA-Z0-9-][a-zA-Z0-9]).)([A-Za-z0-9-][A-Za-z0-9]))*:[0-9]{1,5}/kms$</code>)
location_arnstringThe ARN of the HDFS location. (pattern: <code>^arn:(aws|aws-cn|aws-us-gov|aws-eusc|aws-iso|aws-iso-b):datasync:[a-z-0-9]+:[0-9]{12}:location/loc-[0-9a-z]{17}$</code>)
location_uristringThe URI of the HDFS location. (pattern: <code>^(efs|nfs|s3|smb|hdfs|fsx[a-z0-9-]+):​//[a-zA-Z0-9.:/-]+$</code>)
managed_secret_configobjectDescribes configuration information for a DataSync-managed secret, such as a KerberosKeytab that DataSync uses to access a specific storage location. DataSync uses the default Amazon Web Services-managed KMS key to encrypt this secret in Secrets Manager.
name_nodesarrayThe NameNode that manages the HDFS namespace.
qop_configurationobjectThe Quality of Protection (QOP) configuration, which specifies the Remote Procedure Call (RPC) and data transfer protection settings configured on the HDFS cluster.
replication_factorintegerThe number of DataNodes to replicate the data to when writing to the HDFS cluster.
simple_userstringThe user name to identify the client on the host operating system. This parameter is used if the AuthenticationType is defined as SIMPLE. (pattern: <code>^[.A-Za-z0-9][-.A-Za-z0-9]*$</code>)

Methods​

The following methods are available for this resource:

NameAccessible byRequired ParamsOptional ParamsDescription
describe_location_hdfsselectregionProvides details about how an DataSync transfer location for a Hadoop Distributed File System (HDFS) is configured.
create_location_hdfsinsertregion, NameNodes, AuthenticationType, AgentArnsCreates a transfer location for a Hadoop Distributed File System (HDFS). DataSync can use this location as a source or destination for transferring data. Before you begin, make sure that you understand how DataSync accesses HDFS clusters.
update_location_hdfsupdateregion, LocationArnModifies the following configuration parameters of the Hadoop Distributed File System (HDFS) transfer location that you're using with DataSync. For more information, see Configuring DataSync transfers with an HDFS cluster.

Parameters​

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

NameDatatypeDescription
regionstringAWS region (default: us-east-1)

SELECT examples​

Provides details about how an DataSync transfer location for a Hadoop Distributed File System (HDFS) is configured.

SELECT
agent_arns,
authentication_type,
block_size,
cmk_secret_config,
creation_time,
custom_secret_config,
kerberos_principal,
kms_key_provider_uri,
location_arn,
location_uri,
managed_secret_config,
name_nodes,
qop_configuration,
replication_factor,
simple_user
FROM aws.datasync.location_hdfs
WHERE region = '{{ region }}' -- required
;

INSERT examples​

Creates a transfer location for a Hadoop Distributed File System (HDFS). DataSync can use this location as a source or destination for transferring data. Before you begin, make sure that you understand how DataSync accesses HDFS clusters.

INSERT INTO aws.datasync.location_hdfs (
Subdirectory,
NameNodes,
BlockSize,
ReplicationFactor,
KmsKeyProviderUri,
QopConfiguration,
AuthenticationType,
SimpleUser,
KerberosPrincipal,
KerberosKeytab,
KerberosKrb5Conf,
AgentArns,
Tags,
CmkSecretConfig,
CustomSecretConfig,
region
)
SELECT 
'{{ Subdirectory }}',
'{{ NameNodes }}' /* required */,
{{ BlockSize }},
{{ ReplicationFactor }},
'{{ KmsKeyProviderUri }}',
'{{ QopConfiguration }}',
'{{ AuthenticationType }}' /* required */,
'{{ SimpleUser }}',
'{{ KerberosPrincipal }}',
'{{ KerberosKeytab }}',
'{{ KerberosKrb5Conf }}',
'{{ AgentArns }}' /* required */,
'{{ Tags }}',
'{{ CmkSecretConfig }}',
'{{ CustomSecretConfig }}',
'{{ region }}'
RETURNING
location_arn
;

UPDATE examples​

Modifies the following configuration parameters of the Hadoop Distributed File System (HDFS) transfer location that you're using with DataSync. For more information, see Configuring DataSync transfers with an HDFS cluster.

UPDATE aws.datasync.location_hdfs
SET 
LocationArn = '{{ LocationArn }}',
Subdirectory = '{{ Subdirectory }}',
NameNodes = '{{ NameNodes }}',
BlockSize = {{ BlockSize }},
ReplicationFactor = {{ ReplicationFactor }},
KmsKeyProviderUri = '{{ KmsKeyProviderUri }}',
QopConfiguration = '{{ QopConfiguration }}',
AuthenticationType = '{{ AuthenticationType }}',
SimpleUser = '{{ SimpleUser }}',
KerberosPrincipal = '{{ KerberosPrincipal }}',
KerberosKeytab = '{{ KerberosKeytab }}',
KerberosKrb5Conf = '{{ KerberosKrb5Conf }}',
AgentArns = '{{ AgentArns }}',
CmkSecretConfig = '{{ CmkSecretConfig }}',
CustomSecretConfig = '{{ CustomSecretConfig }}'
WHERE 
region = '{{ region }}' --required
AND LocationArn = '{{ LocationArn }}' --required;
Last updated on