sessions
Creates, updates, deletes, gets or lists a sessions resource.
Overview
| Name | sessions |
| Type | Resource |
| Id | aws.glue.sessions |
Fields
The following fields are returned by SELECT queries:
- get_session
| Name | Datatype | Description |
|---|---|---|
command | object | The command object.See SessionCommand. |
completed_on | string (date-time) | The date and time that this session is completed. |
connections | object | The number of connections used for the session. |
created_on | string (date-time) | The time and date when the session was created. |
dpu_seconds | number (double) | The DPUs consumed by the session (formula: ExecutionTime * MaxCapacity). |
default_arguments | object | A map array of key-value pairs. Max is 75 pairs. |
description | string | The description of the session. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>) |
error_message | string | The error message displayed during the session. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*</code>) |
execution_time | number (double) | The total time the session ran for. |
glue_version | string | The Glue version determines the versions of Apache Spark and Python that Glue supports. The GlueVersion must be greater than 2.0. (pattern: <code>^(\w+.)+\w+$</code>) |
id | string | The ID of the session. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>) |
idle_timeout | integer | The number of minutes when idle before the session times out. |
max_capacity | number (double) | The number of Glue data processing units (DPUs) that can be allocated when the job runs. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB memory. |
number_of_workers | integer | The number of workers of a defined WorkerType to use for the session. |
profile_name | string | The name of an Glue usage profile associated with the session. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>) |
progress | number (double) | The code execution progress of the session. |
role | string | The name or Amazon Resource Name (ARN) of the IAM role associated with the Session. (pattern: <code>arn:aws[^:]:iam::[0-9]:role/.+</code>) |
security_configuration | string | The name of the SecurityConfiguration structure to be used with the session. (pattern: <code>[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\t]*</code>) |
session_type | string | The type of the session. (LIVY, SPARK_CONNECT) |
status | string | The session status. (PROVISIONING, READY, FAILED, TIMEOUT, STOPPING, STOPPED) |
worker_type | string | The type of predefined worker that is allocated when a session runs. Accepts a value of G.1X, G.2X, G.4X, or G.8X for Spark sessions. Accepts the value Z.2X for Ray sessions. (Standard, G.1X, G.2X, G.025X, G.4X, G.8X, Z.2X) |
Methods
The following methods are available for this resource:
| Name | Accessible by | Required Params | Optional Params | Description |
|---|---|---|---|---|
get_session | select | region | Retrieves the session. | |
create_session | insert | region, Role, Command | Creates a new session. | |
delete_session | delete | region | Deletes the session. | |
cancel_statement | exec | region, SessionId | Cancels the statement. | |
list_sessions | exec | region | Retrieve a list of sessions. | |
run_statement | exec | region, SessionId, Code | Executes the statement. |
Parameters
Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.
| Name | Datatype | Description |
|---|---|---|
region | string | AWS region (default: us-east-1) |
SELECT examples
- get_session
Retrieves the session.
SELECT
command,
completed_on,
connections,
created_on,
dpu_seconds,
default_arguments,
description,
error_message,
execution_time,
glue_version,
id,
idle_timeout,
max_capacity,
number_of_workers,
profile_name,
progress,
role,
security_configuration,
session_type,
status,
worker_type
FROM aws.glue.sessions
WHERE region = '{{ region }}' -- required
;
INSERT examples
- create_session
- Manifest
Creates a new session.
INSERT INTO aws.glue.sessions (
Id,
Description,
Role,
Command,
Timeout,
IdleTimeout,
DefaultArguments,
Connections,
MaxCapacity,
NumberOfWorkers,
WorkerType,
SecurityConfiguration,
GlueVersion,
Tags,
RequestOrigin,
SessionType,
region
)
SELECT
'{{ Id }}',
'{{ Description }}',
'{{ Role }}' /* required */,
'{{ Command }}' /* required */,
{{ Timeout }},
{{ IdleTimeout }},
'{{ DefaultArguments }}',
'{{ Connections }}',
{{ MaxCapacity }},
{{ NumberOfWorkers }},
'{{ WorkerType }}',
'{{ SecurityConfiguration }}',
'{{ GlueVersion }}',
'{{ Tags }}',
'{{ RequestOrigin }}',
'{{ SessionType }}',
'{{ region }}'
RETURNING
session
;
# Description fields are for documentation purposes
- name: sessions
props:
- name: region
value: "{{ region }}"
description: Required parameter for the sessions resource.
- name: Id
value: "{{ Id }}"
description: |
The ID of the session request.
- name: Description
value: "{{ Description }}"
description: |
The description of the session.
- name: Role
value: "{{ Role }}"
description: |
The IAM Role ARN
- name: Command
description: |
The SessionCommand that runs the job.
value:
Name: "{{ Name }}"
PythonVersion: "{{ PythonVersion }}"
- name: Timeout
value: {{ Timeout }}
description: |
The number of minutes before session times out. Default for Spark ETL jobs is 48 hours (2880 minutes). Consult the documentation for other job types.
- name: IdleTimeout
value: {{ IdleTimeout }}
description: |
The number of minutes when idle before session times out. Default for Spark ETL jobs is value of Timeout. Consult the documentation for other job types.
- name: DefaultArguments
value: "{{ DefaultArguments }}"
description: |
A map array of key-value pairs. Max is 75 pairs.
- name: Connections
description: |
The number of connections to use for the session.
value:
Connections:
- "{{ Connections }}"
- name: MaxCapacity
value: {{ MaxCapacity }}
description: |
The number of Glue data processing units (DPUs) that can be allocated when the job runs. A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB memory.
- name: NumberOfWorkers
value: {{ NumberOfWorkers }}
description: |
The number of workers of a defined WorkerType to use for the session.
- name: WorkerType
value: "{{ WorkerType }}"
description: |
The type of predefined worker that is allocated when a job runs. Accepts a value of G.1X, G.2X, G.4X, or G.8X for Spark jobs. Accepts the value Z.2X for Ray notebooks. For the G.1X worker type, each worker maps to 1 DPU (4 vCPUs, 16 GB of memory) with 94GB disk, and provides 1 executor per worker. We recommend this worker type for workloads such as data transforms, joins, and queries, to offers a scalable and cost effective way to run most jobs. For the G.2X worker type, each worker maps to 2 DPU (8 vCPUs, 32 GB of memory) with 138GB disk, and provides 1 executor per worker. We recommend this worker type for workloads such as data transforms, joins, and queries, to offers a scalable and cost effective way to run most jobs. For the G.4X worker type, each worker maps to 4 DPU (16 vCPUs, 64 GB of memory) with 256GB disk, and provides 1 executor per worker. We recommend this worker type for jobs whose workloads contain your most demanding transforms, aggregations, joins, and queries. This worker type is available only for Glue version 3.0 or later Spark ETL jobs in the following Amazon Web Services Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm). For the G.8X worker type, each worker maps to 8 DPU (32 vCPUs, 128 GB of memory) with 512GB disk, and provides 1 executor per worker. We recommend this worker type for jobs whose workloads contain your most demanding transforms, aggregations, joins, and queries. This worker type is available only for Glue version 3.0 or later Spark ETL jobs, in the same Amazon Web Services Regions as supported for the G.4X worker type. For the Z.2X worker type, each worker maps to 2 M-DPU (8vCPUs, 64 GB of memory) with 128 GB disk, and provides up to 8 Ray workers based on the autoscaler.
valid_values: ['Standard', 'G.1X', 'G.2X', 'G.025X', 'G.4X', 'G.8X', 'Z.2X']
- name: SecurityConfiguration
value: "{{ SecurityConfiguration }}"
description: |
The name of the SecurityConfiguration structure to be used with the session
- name: GlueVersion
value: "{{ GlueVersion }}"
description: |
The Glue version determines the versions of Apache Spark and Python that Glue supports. The GlueVersion must be greater than 2.0.
- name: Tags
value: "{{ Tags }}"
description: |
The map of key value pairs (tags) belonging to the session.
- name: RequestOrigin
value: "{{ RequestOrigin }}"
description: |
The origin of the request.
- name: SessionType
value: "{{ SessionType }}"
description: |
The type of session to create.
valid_values: ['LIVY', 'SPARK_CONNECT']
DELETE examples
- delete_session
Deletes the session.
DELETE FROM aws.glue.sessions
WHERE region = '{{ region }}' --required
;
Lifecycle Methods
- cancel_statement
- list_sessions
- run_statement
Cancels the statement.
EXEC aws.glue.sessions.cancel_statement
@region='{{ region }}' --required
@@json=
'{
"SessionId": "{{ SessionId }}",
"Id": {{ Id }},
"RequestOrigin": "{{ RequestOrigin }}"
}'
;
Retrieve a list of sessions.
EXEC aws.glue.sessions.list_sessions
@region='{{ region }}' --required
@@json=
'{
"NextToken": "{{ NextToken }}",
"MaxResults": {{ MaxResults }},
"Tags": "{{ Tags }}",
"RequestOrigin": "{{ RequestOrigin }}"
}'
;
Executes the statement.
EXEC aws.glue.sessions.run_statement
@region='{{ region }}' --required
@@json=
'{
"SessionId": "{{ SessionId }}",
"Code": "{{ Code }}",
"RequestOrigin": "{{ RequestOrigin }}"
}'
;