evaluators

Creates, updates, deletes, gets or lists an evaluators resource.

Overview

Name	evaluators
Type	Resource
Id	aws.bedrock_agentcore_control.evaluators

Fields

The following fields are returned by SELECT queries:

get_evaluator

Name	Datatype	Description
created_at	string (date-time)	The timestamp when the evaluator was created.
description	string	The description of the evaluator.
evaluator_arn	string	The Amazon Resource Name (ARN) of the evaluator. (pattern: <code>arn:aws[a-zA-Z-]:bedrock-agentcore:[a-z0-9-]+:[0-9]{12}:evaluator/[a-zA-Z][a-zA-Z0-9-_]{0,99}-[a-zA-Z0-9]{10}$|^arn:aws[a-zA-Z-]:bedrock-agentcore:::evaluator/(Builtin|ThirdParty).[a-zA-Z0-9._-]+</code>)
evaluator_config	object	The configuration that defines how an evaluator assesses agent performance, including the evaluation method and parameters.
evaluator_id	string	The unique identifier of the evaluator. (pattern: <code>(Builtin.[a-zA-Z0-9.-]+|ThirdParty.[a-zA-Z0-9-]+.[a-zA-Z0-9_-]+|[a-zA-Z][a-zA-Z0-9-_]{0,99}-[a-zA-Z0-9]{10})</code>)
evaluator_name	string	The name of the evaluator. (pattern: <code>(Builtin.[a-zA-Z0-9.-]+|ThirdParty.[a-zA-Z0-9-]+.[a-zA-Z0-9_-]+|[a-zA-Z][a-zA-Z0-9_]{0,47})</code>)
evaluator_type	string	The kind of evaluator resource. Valid values: Builtin – An Amazon Web Services-managed global evaluator. ThirdParty – An Amazon Web Services-managed global evaluator from a third-party provider. Custom – A customer-created evaluator. CustomCode – A customer-created code-based evaluator. CustomDerived – A customer-created evaluator derived from an existing base evaluator. (Builtin, ThirdParty, Custom, CustomCode, CustomDerived)
kms_key_arn	string	The Amazon Resource Name (ARN) of the customer managed KMS key used to encrypt the evaluator's sensitive data. This field is only present for evaluators encrypted with a customer managed key. (pattern: <code>arn:aws(|-cn|-us-gov):kms:[a-zA-Z0-9-]*:[0-9]{12}:key/[a-zA-Z0-9-]{36}</code>)
level	string	The evaluation level (TOOL_CALL, TRACE, or SESSION) that determines the scope of evaluation. (TOOL_CALL, TRACE, SESSION)
locked_for_modification	boolean	Whether the evaluator is locked for modification due to being referenced by active online evaluation configurations.
provider	string	The source of the evaluator's logic: Amazon Web Services, a third-party library, or you. (AWS, DeepEval, AutoEval, Custom)
status	string	The current status of the evaluator. (ACTIVE, CREATING, CREATE_FAILED, UPDATING, UPDATE_FAILED, DELETING)
updated_at	string (date-time)	The timestamp when the evaluator was last updated.

list_evaluators

Name	Datatype	Description
created_at	string (date-time)	The timestamp when the evaluator was created.
description	string	The description of the evaluator.
evaluator_arn	string	The Amazon Resource Name (ARN) of the evaluator. (pattern: <code>arn:aws[a-zA-Z-]:bedrock-agentcore:[a-z0-9-]+:[0-9]{12}:evaluator/[a-zA-Z][a-zA-Z0-9-_]{0,99}-[a-zA-Z0-9]{10}$|^arn:aws[a-zA-Z-]:bedrock-agentcore:::evaluator/(Builtin|ThirdParty).[a-zA-Z0-9._-]+</code>)
evaluator_id	string	The unique identifier of the evaluator. (pattern: <code>(Builtin.[a-zA-Z0-9.-]+|ThirdParty.[a-zA-Z0-9-]+.[a-zA-Z0-9_-]+|[a-zA-Z][a-zA-Z0-9-_]{0,99}-[a-zA-Z0-9]{10})</code>)
evaluator_name	string	The name of the evaluator. (pattern: <code>(Builtin.[a-zA-Z0-9.-]+|ThirdParty.[a-zA-Z0-9-]+.[a-zA-Z0-9_-]+|[a-zA-Z][a-zA-Z0-9_]{0,47})</code>)
evaluator_type	string	The type of evaluator, indicating whether it is a built-in evaluator provided by the service or a custom evaluator created by the user. (Builtin, ThirdParty, Custom, CustomCode, CustomDerived)
kms_key_arn	string	The Amazon Resource Name (ARN) of the customer managed KMS key used to encrypt the evaluator's sensitive data. This field is only present for evaluators encrypted with a customer managed key. (pattern: <code>arn:aws(|-cn|-us-gov):kms:[a-zA-Z0-9-]*:[0-9]{12}:key/[a-zA-Z0-9-]{36}</code>)
level	string	The evaluation level (TOOL_CALL, TRACE, or SESSION) that determines the scope of evaluation. (TOOL_CALL, TRACE, SESSION)
locked_for_modification	boolean	Whether the evaluator is locked for modification due to being referenced by active online evaluation configurations.
provider	string	The source of the evaluator's logic: Amazon Web Services, a third-party library, or you. (AWS, DeepEval, AutoEval, Custom)
status	string	The current status of the evaluator. (ACTIVE, CREATING, CREATE_FAILED, UPDATING, UPDATE_FAILED, DELETING)
updated_at	string (date-time)	The timestamp when the evaluator was last updated.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_evaluator	select	evaluator_id, region	includedData	Retrieves detailed information about an evaluator, including its configuration, status, and metadata. Works with both built-in and custom evaluators.
list_evaluators	select	region	nextToken, maxResults	Lists all available evaluators, including both builtin evaluators provided by the service and custom evaluators created by the user.
create_evaluator	insert	region, evaluatorName, evaluatorConfig, level		Creates a custom evaluator for agent quality assessment. Custom evaluators can use either LLM-as-a-Judge configurations with user-defined prompts, rating scales, and model settings, or code-based configurations with customer-managed Lambda functions to evaluate agent performance at tool call, trace, or session levels.
update_evaluator	update	evaluator_id, region		Updates a custom evaluator's configuration, description, or evaluation level. Built-in evaluators cannot be updated. The evaluator must not be locked for modification.
delete_evaluator	delete	evaluator_id, region		Deletes a custom evaluator. Builtin evaluators cannot be deleted. The evaluator must not be referenced by any active online evaluation configurations.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
evaluator_id	string	The unique identifier of the evaluator to delete.
region	string	AWS region (default: us-east-1)
includedData	string	Controls which data is returned in the response. ALL_DATA (default) returns the full evaluator including decrypted instructions and rating scale. For evaluators encrypted with a customer managed KMS key, this requires kms:Decrypt permission on the key. METADATA_ONLY returns evaluator metadata and model configuration without instructions or rating scale, and does not require any KMS permissions.
maxResults	integer	The maximum number of evaluators to return in a single response.
nextToken	string	The pagination token from a previous request to retrieve the next page of results.

SELECT examples

get_evaluator

Retrieves detailed information about an evaluator, including its configuration, status, and metadata. Works with both built-in and custom evaluators.

SELECT
created_at,
description,
evaluator_arn,
evaluator_config,
evaluator_id,
evaluator_name,
evaluator_type,
kms_key_arn,
level,
locked_for_modification,
provider,
status,
updated_at
FROM aws.bedrock_agentcore_control.evaluators
WHERE evaluator_id = '{{ evaluator_id }}' -- required
AND region = '{{ region }}' -- required
AND includedData = '{{ includedData }}'
;

list_evaluators

Lists all available evaluators, including both builtin evaluators provided by the service and custom evaluators created by the user.

SELECT
created_at,
description,
evaluator_arn,
evaluator_id,
evaluator_name,
evaluator_type,
kms_key_arn,
level,
locked_for_modification,
provider,
status,
updated_at
FROM aws.bedrock_agentcore_control.evaluators
WHERE region = '{{ region }}' -- required
AND nextToken = '{{ nextToken }}'
AND maxResults = '{{ maxResults }}'
;

INSERT examples

create_evaluator

Creates a custom evaluator for agent quality assessment. Custom evaluators can use either LLM-as-a-Judge configurations with user-defined prompts, rating scales, and model settings, or code-based configurations with customer-managed Lambda functions to evaluate agent performance at tool call, trace, or session levels.

INSERT INTO aws.bedrock_agentcore_control.evaluators (
clientToken,
evaluatorName,
description,
evaluatorConfig,
level,
kmsKeyArn,
tags,
region
)
SELECT 
'{{ clientToken }}',
'{{ evaluatorName }}' /* required */,
'{{ description }}',
'{{ evaluatorConfig }}' /* required */,
'{{ level }}' /* required */,
'{{ kmsKeyArn }}',
'{{ tags }}',
'{{ region }}'
RETURNING
created_at,
evaluator_arn,
evaluator_id,
status
;

Manifest

# Description fields are for documentation purposes
- name: evaluators
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the evaluators resource.
  - name: clientToken
    value: "{{ clientToken }}"
  - name: evaluatorName
    value: "{{ evaluatorName }}"
  - name: description
    value: "{{ description }}"
  - name: evaluatorConfig
    description: |
      The configuration that defines how an evaluator assesses agent performance, including the evaluation method and parameters.
    value:
      llmAsAJudge:
        instructions: "{{ instructions }}"
        ratingScale:
          numerical:
            - definition: "{{ definition }}"
              value: {{ value }}
              label: "{{ label }}"
          categorical:
            - definition: "{{ definition }}"
              label: "{{ label }}"
        modelConfig:
          bedrockEvaluatorModelConfig:
            modelId: "{{ modelId }}"
            inferenceConfig:
              maxTokens: {{ maxTokens }}
              temperature: {{ temperature }}
              topP: {{ topP }}
              stopSequences: "{{ stopSequences }}"
            additionalModelRequestFields: "{{ additionalModelRequestFields }}"
          responsesEvaluatorModelConfig:
            modelId: "{{ modelId }}"
            maxOutputTokens: {{ maxOutputTokens }}
            temperature: {{ temperature }}
            topP: {{ topP }}
            reasoning:
              effort: "{{ effort }}"
      codeBased:
        lambdaConfig:
          lambdaArn: "{{ lambdaArn }}"
          lambdaTimeoutInSeconds: {{ lambdaTimeoutInSeconds }}
      derived:
        baseEvaluatorId: "{{ baseEvaluatorId }}"
        modelConfig:
          bedrockEvaluatorModelConfig:
            modelId: "{{ modelId }}"
            inferenceConfig:
              maxTokens: {{ maxTokens }}
              temperature: {{ temperature }}
              topP: {{ topP }}
              stopSequences: "{{ stopSequences }}"
            additionalModelRequestFields: "{{ additionalModelRequestFields }}"
          responsesEvaluatorModelConfig:
            modelId: "{{ modelId }}"
            maxOutputTokens: {{ maxOutputTokens }}
            temperature: {{ temperature }}
            topP: {{ topP }}
            reasoning:
              effort: "{{ effort }}"
  - name: level
    value: "{{ level }}"
    valid_values: ['TOOL_CALL', 'TRACE', 'SESSION']
  - name: kmsKeyArn
    value: "{{ kmsKeyArn }}"
  - name: tags
    value: "{{ tags }}"

UPDATE examples

update_evaluator

Updates a custom evaluator's configuration, description, or evaluation level. Built-in evaluators cannot be updated. The evaluator must not be locked for modification.

UPDATE aws.bedrock_agentcore_control.evaluators
SET 
clientToken = '{{ clientToken }}',
description = '{{ description }}',
evaluatorConfig = '{{ evaluatorConfig }}',
level = '{{ level }}',
kmsKeyArn = '{{ kmsKeyArn }}'
WHERE 
evaluator_id = '{{ evaluator_id }}' --required
AND region = '{{ region }}' --required
RETURNING
evaluator_arn,
evaluator_id,
status,
updated_at;

DELETE examples

delete_evaluator

Deletes a custom evaluator. Builtin evaluators cannot be deleted. The evaluator must not be referenced by any active online evaluation configurations.

DELETE FROM aws.bedrock_agentcore_control.evaluators
WHERE evaluator_id = '{{ evaluator_id }}' --required
AND region = '{{ region }}' --required
;