kx_clusters

Creates, updates, deletes, gets or lists a kx_clusters resource.

Overview

Name	`kx_clusters`
Type	Resource
Id	`aws.finspace.kx_clusters`

Fields

The following fields are returned by SELECT queries:

get_kx_cluster
list_kx_clusters

Name	Datatype	Description
`auto_scaling_configuration`	`object`	The configuration based on which FinSpace will scale in or scale out nodes in your cluster.
`availability_zone_id`	`string`	The availability zone identifiers for the requested regions. (pattern: <code>^[a-zA-Z0-9-]+$</code>)
`az_mode`	`string`	The number of availability zones you want to assign per cluster. This can be one of the following SINGLE – Assigns one availability zone per cluster. MULTI – Assigns all the availability zones per cluster. (SINGLE, MULTI)
`cache_storage_configurations`	`array`	The configurations for a read only cache storage associated with a cluster. This cache will be stored as an FSx Lustre that reads from the S3 store.
`capacity_configuration`	`object`	A structure for the metadata of a cluster. It includes information like the CPUs needed, memory of instances, and number of instances.
`cluster_description`	`string`	A description of the cluster. (pattern: <code>^[a-zA-Z0-9_-.\s]+$</code>)
`cluster_name`	`string`	A unique name for the cluster. (pattern: <code>^[a-zA-Z0-9][a-zA-Z0-9-_]*[a-zA-Z0-9]$</code>)
`cluster_type`	`string`	Specifies the type of KDB database that is being created. The following types are available: HDB – A Historical Database. The data is only accessible with read-only permissions from one of the FinSpace managed kdb databases mounted to the cluster. RDB – A Realtime Database. This type of database captures all the data from a ticker plant and stores it in memory until the end of day, after which it writes all of its data to a disk and reloads the HDB. This cluster type requires local storage for temporary storage of data during the savedown process. If you specify this field in your request, you must provide the savedownStorageConfiguration parameter. GATEWAY – A gateway cluster allows you to access data across processes in kdb systems. It allows you to create your own routing logic using the initialization scripts and custom code. This type of cluster does not require a writable local storage. GP – A general purpose cluster allows you to quickly iterate on code during development by granting greater access to system commands and enabling a fast reload of custom code. This cluster type can optionally mount databases including cache and savedown storage. For this cluster type, the node count is fixed at 1. It does not support autoscaling and supports only SINGLE AZ mode. Tickerplant – A tickerplant cluster allows you to subscribe to feed handlers based on IAM permissions. It can publish to RDBs, other Tickerplants, and real-time subscribers (RTS). Tickerplants can persist messages to log, which is readable by any RDB environment. It supports only single-node that is only one kdb process. (HDB, RDB, GATEWAY, GP, TICKERPLANT)
`code`	`object`	The structure of the customer code available within the running cluster.
`command_line_arguments`	`array`	Defines key-value pairs to make them available inside the cluster.
`created_timestamp`	`string (date-time)`	The timestamp at which the cluster was created in FinSpace. The value is determined as epoch time in milliseconds. For example, the value for Monday, November 1, 2021 12:00:00 PM UTC is specified as 1635768000000.
`databases`	`array`	A list of databases mounted on the cluster.
`execution_role`	`string`	An IAM role that defines a set of permissions associated with a cluster. These permissions are assumed when a cluster attempts to access another cluster. (pattern: <code>^arn:aws[a-z0-9-]*:iam::\d{12}:role/[\w-/.@+=,]{1,1017}$</code>)
`initialization_script`	`string`	Specifies a Q program that will be run at launch of a cluster. It is a relative path within .zip file that contains the custom code, which will be loaded on the cluster. It must include the file name itself. For example, somedir/init.q. (pattern: <code>^[a-zA-Z0-9_-./\]+$</code>)
`last_modified_timestamp`	`string (date-time)`	The last time that the cluster was modified. The value is determined as epoch time in milliseconds. For example, the value for Monday, November 1, 2021 12:00:00 PM UTC is specified as 1635768000000.
`release_label`	`string`	The version of FinSpace managed kdb to run. (pattern: <code>^[a-zA-Z0-9._-]+$</code>)
`savedown_storage_configuration`	`object`	The size and type of temporary storage that is used to hold data during the savedown process. All the data written to this storage space is lost when the cluster node is restarted.
`scaling_group_configuration`	`object`	The structure that stores the capacity configuration details of a scaling group.
`status`	`string`	The status of cluster creation. PENDING – The cluster is pending creation. CREATING – The cluster creation process is in progress. CREATE_FAILED – The cluster creation process has failed. RUNNING – The cluster creation process is running. UPDATING – The cluster is in the process of being updated. DELETING – The cluster is in the process of being deleted. DELETED – The cluster has been deleted. DELETE_FAILED – The cluster failed to delete. (PENDING, CREATING, CREATE_FAILED, RUNNING, UPDATING, DELETING, DELETED, DELETE_FAILED)
`status_reason`	`string`	The error message when a failed state occurs. (pattern: <code>^[a-zA-Z0-9_-.\s]+$</code>)
`tickerplant_log_configuration`	`object`	A configuration to store the Tickerplant logs. It consists of a list of volumes that will be mounted to your cluster. For the cluster type Tickerplant, the location of the TP volume on the cluster will be available by using the global variable .aws.tp_log_path.
`volumes`	`array`	A list of volumes attached to the cluster.
`vpc_configuration`	`object`	Configuration details about the network where the Privatelink endpoint of the cluster resides.

Name	Datatype	Description
`kx_cluster_summaries`	`array`	Lists the cluster details.
`next_token`	`string`	A token that indicates where a results page should begin. (pattern: <code>.*</code>)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
`get_kx_cluster`	`select`	`environment_id`, `cluster_name`, `region`		Retrieves information about a kdb cluster.
`list_kx_clusters`	`select`	`environment_id`, `region`	`clusterType`, `maxResults`, `nextToken`	Returns a list of clusters.
`create_kx_cluster`	`insert`	`environment_id`, `region`, `clusterName`, `clusterType`, `releaseLabel`, `vpcConfiguration`, `azMode`		Creates a new kdb cluster.
`update_kx_cluster_code_configuration`	`update`	`environment_id`, `cluster_name`, `region`, `code`		Allows you to update code configuration on a running cluster. By using this API you can update the code, the initialization script path, and the command line arguments for a specific cluster. The configuration that you want to update will override any existing configurations on the cluster.
`update_kx_cluster_databases`	`update`	`environment_id`, `cluster_name`, `region`, `databases`		Updates the databases mounted on a kdb cluster, which includes the changesetId and all the dbPaths to be cached. This API does not allow you to change a database name or add a database if you created a cluster without one. Using this API you can point a cluster to a different changeset and modify a list of partitions being cached.
`delete_kx_cluster`	`delete`	`environment_id`, `cluster_name`, `region`	`clientToken`	Deletes a kdb cluster.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
`cluster_name`	`string`	The name of the cluster that you want to delete.
`environment_id`	`string`	A unique identifier for the kdb environment.
`region`	`string`	AWS region (default: us-east-1)
`clientToken`	`string`	A token that ensures idempotency. This token expires in 10 minutes.
`clusterType`	`string`	Specifies the type of KDB database that is being created. The following types are available: HDB – A Historical Database. The data is only accessible with read-only permissions from one of the FinSpace managed kdb databases mounted to the cluster. RDB – A Realtime Database. This type of database captures all the data from a ticker plant and stores it in memory until the end of day, after which it writes all of its data to a disk and reloads the HDB. This cluster type requires local storage for temporary storage of data during the savedown process. If you specify this field in your request, you must provide the savedownStorageConfiguration parameter. GATEWAY – A gateway cluster allows you to access data across processes in kdb systems. It allows you to create your own routing logic using the initialization scripts and custom code. This type of cluster does not require a writable local storage. GP – A general purpose cluster allows you to quickly iterate on code during development by granting greater access to system commands and enabling a fast reload of custom code. This cluster type can optionally mount databases including cache and savedown storage. For this cluster type, the node count is fixed at 1. It does not support autoscaling and supports only SINGLE AZ mode. Tickerplant – A tickerplant cluster allows you to subscribe to feed handlers based on IAM permissions. It can publish to RDBs, other Tickerplants, and real-time subscribers (RTS). Tickerplants can persist messages to log, which is readable by any RDB environment. It supports only single-node that is only one kdb process.
`maxResults`	`integer`	The maximum number of results to return in this request.
`nextToken`	`string`	A token that indicates where a results page should begin.

`SELECT` examples

get_kx_cluster
list_kx_clusters

Retrieves information about a kdb cluster.

SELECT
auto_scaling_configuration,
availability_zone_id,
az_mode,
cache_storage_configurations,
capacity_configuration,
cluster_description,
cluster_name,
cluster_type,
code,
command_line_arguments,
created_timestamp,
databases,
execution_role,
initialization_script,
last_modified_timestamp,
release_label,
savedown_storage_configuration,
scaling_group_configuration,
status,
status_reason,
tickerplant_log_configuration,
volumes,
vpc_configuration
FROM aws.finspace.kx_clusters
WHERE environment_id = '{{ environment_id }}' -- required
AND cluster_name = '{{ cluster_name }}' -- required
AND region = '{{ region }}' -- required
;

Returns a list of clusters.

SELECT
kx_cluster_summaries,
next_token
FROM aws.finspace.kx_clusters
WHERE environment_id = '{{ environment_id }}' -- required
AND region = '{{ region }}' -- required
AND clusterType = '{{ clusterType }}'
AND maxResults = '{{ maxResults }}'
AND nextToken = '{{ nextToken }}'
;

`INSERT` examples

create_kx_cluster
Manifest

Creates a new kdb cluster.

INSERT INTO aws.finspace.kx_clusters (
clientToken,
clusterName,
clusterType,
tickerplantLogConfiguration,
databases,
cacheStorageConfigurations,
autoScalingConfiguration,
clusterDescription,
capacityConfiguration,
releaseLabel,
vpcConfiguration,
initializationScript,
commandLineArguments,
code,
executionRole,
savedownStorageConfiguration,
azMode,
availabilityZoneId,
tags,
scalingGroupConfiguration,
environment_id,
region
)
SELECT 
'{{ clientToken }}',
'{{ clusterName }}' /* required */,
'{{ clusterType }}' /* required */,
'{{ tickerplantLogConfiguration }}',
'{{ databases }}',
'{{ cacheStorageConfigurations }}',
'{{ autoScalingConfiguration }}',
'{{ clusterDescription }}',
'{{ capacityConfiguration }}',
'{{ releaseLabel }}' /* required */,
'{{ vpcConfiguration }}' /* required */,
'{{ initializationScript }}',
'{{ commandLineArguments }}',
'{{ code }}',
'{{ executionRole }}',
'{{ savedownStorageConfiguration }}',
'{{ azMode }}' /* required */,
'{{ availabilityZoneId }}',
'{{ tags }}',
'{{ scalingGroupConfiguration }}',
'{{ environment_id }}',
'{{ region }}'
RETURNING
auto_scaling_configuration,
availability_zone_id,
az_mode,
cache_storage_configurations,
capacity_configuration,
cluster_description,
cluster_name,
cluster_type,
code,
command_line_arguments,
created_timestamp,
databases,
environment_id,
execution_role,
initialization_script,
last_modified_timestamp,
release_label,
savedown_storage_configuration,
scaling_group_configuration,
status,
status_reason,
tickerplant_log_configuration,
volumes,
vpc_configuration
;

# Description fields are for documentation purposes
- name: kx_clusters
props:
  - name: environment_id
    value: "{{ environment_id }}"
    description: Required parameter for the kx_clusters resource.
  - name: region
    value: "{{ region }}"
    description: Required parameter for the kx_clusters resource.
  - name: clientToken
    value: "{{ clientToken }}"
  - name: clusterName
    value: "{{ clusterName }}"
  - name: clusterType
    value: "{{ clusterType }}"
    valid_values: ['HDB', 'RDB', 'GATEWAY', 'GP', 'TICKERPLANT']
  - name: tickerplantLogConfiguration
    description: |
      A configuration to store the Tickerplant logs. It consists of a list of volumes that will be mounted to your cluster. For the cluster type Tickerplant, the location of the TP volume on the cluster will be available by using the global variable .aws.tp_log_path.
    value:
      tickerplantLogVolumes:
        - "{{ tickerplantLogVolumes }}"
  - name: databases
    value:
      - databaseName: "{{ databaseName }}"
        cacheConfigurations: "{{ cacheConfigurations }}"
        changesetId: "{{ changesetId }}"
        dataviewName: "{{ dataviewName }}"
        dataviewConfiguration:
          dataviewName: "{{ dataviewName }}"
          dataviewVersionId: "{{ dataviewVersionId }}"
          changesetId: "{{ changesetId }}"
          segmentConfigurations:
            - dbPaths: "{{ dbPaths }}"
              volumeName: "{{ volumeName }}"
              onDemand: {{ onDemand }}
  - name: cacheStorageConfigurations
    value:
      - type_: "{{ type_ }}"
        size: {{ size }}
  - name: autoScalingConfiguration
    description: |
      The configuration based on which FinSpace will scale in or scale out nodes in your cluster.
    value:
      minNodeCount: {{ minNodeCount }}
      maxNodeCount: {{ maxNodeCount }}
      autoScalingMetric: "{{ autoScalingMetric }}"
      metricTarget: {{ metricTarget }}
      scaleInCooldownSeconds: {{ scaleInCooldownSeconds }}
      scaleOutCooldownSeconds: {{ scaleOutCooldownSeconds }}
  - name: clusterDescription
    value: "{{ clusterDescription }}"
  - name: capacityConfiguration
    description: |
      A structure for the metadata of a cluster. It includes information like the CPUs needed, memory of instances, and number of instances.
    value:
      nodeType: "{{ nodeType }}"
      nodeCount: {{ nodeCount }}
  - name: releaseLabel
    value: "{{ releaseLabel }}"
  - name: vpcConfiguration
    description: |
      Configuration details about the network where the Privatelink endpoint of the cluster resides.
    value:
      vpcId: "{{ vpcId }}"
      securityGroupIds:
        - "{{ securityGroupIds }}"
      subnetIds:
        - "{{ subnetIds }}"
      ipAddressType: "{{ ipAddressType }}"
  - name: initializationScript
    value: "{{ initializationScript }}"
  - name: commandLineArguments
    value:
      - key: "{{ key }}"
        value: "{{ value }}"
  - name: code
    description: |
      The structure of the customer code available within the running cluster.
    value:
      s3Bucket: "{{ s3Bucket }}"
      s3Key: "{{ s3Key }}"
      s3ObjectVersion: "{{ s3ObjectVersion }}"
  - name: executionRole
    value: "{{ executionRole }}"
  - name: savedownStorageConfiguration
    description: |
      The size and type of temporary storage that is used to hold data during the savedown process. All the data written to this storage space is lost when the cluster node is restarted.
    value:
      type_: "{{ type_ }}"
      size: {{ size }}
      volumeName: "{{ volumeName }}"
  - name: azMode
    value: "{{ azMode }}"
    valid_values: ['SINGLE', 'MULTI']
  - name: availabilityZoneId
    value: "{{ availabilityZoneId }}"
  - name: tags
    value: "{{ tags }}"
  - name: scalingGroupConfiguration
    description: |
      The structure that stores the capacity configuration details of a scaling group.
    value:
      scalingGroupName: "{{ scalingGroupName }}"
      memoryLimit: {{ memoryLimit }}
      memoryReservation: {{ memoryReservation }}
      nodeCount: {{ nodeCount }}
      cpu: {{ cpu }}

`UPDATE` examples

update_kx_cluster_code_configuration
update_kx_cluster_databases

Allows you to update code configuration on a running cluster. By using this API you can update the code, the initialization script path, and the command line arguments for a specific cluster. The configuration that you want to update will override any existing configurations on the cluster.

UPDATE aws.finspace.kx_clusters
SET 
clientToken = '{{ clientToken }}',
code = '{{ code }}',
initializationScript = '{{ initializationScript }}',
commandLineArguments = '{{ commandLineArguments }}',
deploymentConfiguration = '{{ deploymentConfiguration }}'
WHERE 
environment_id = '{{ environment_id }}' --required
AND cluster_name = '{{ cluster_name }}' --required
AND region = '{{ region }}' --required
AND code = '{{ code }}' --required;

Updates the databases mounted on a kdb cluster, which includes the changesetId and all the dbPaths to be cached. This API does not allow you to change a database name or add a database if you created a cluster without one. Using this API you can point a cluster to a different changeset and modify a list of partitions being cached.

UPDATE aws.finspace.kx_clusters
SET 
clientToken = '{{ clientToken }}',
databases = '{{ databases }}',
deploymentConfiguration = '{{ deploymentConfiguration }}'
WHERE 
environment_id = '{{ environment_id }}' --required
AND cluster_name = '{{ cluster_name }}' --required
AND region = '{{ region }}' --required
AND databases = '{{ databases }}' --required;

`DELETE` examples

delete_kx_cluster

Deletes a kdb cluster.

DELETE FROM aws.finspace.kx_clusters
WHERE environment_id = '{{ environment_id }}' --required
AND cluster_name = '{{ cluster_name }}' --required
AND region = '{{ region }}' --required
AND clientToken = '{{ clientToken }}'
;

Overview​

Fields​

Methods​

Parameters​

SELECT examples​

INSERT examples​

UPDATE examples​

DELETE examples​