evaluation_jobs

Creates, updates, deletes, gets or lists an evaluation_jobs resource.

Overview

Name	evaluation_jobs
Type	Resource
Id	aws.bedrock.evaluation_jobs

Fields

The following fields are returned by SELECT queries:

get_evaluation_job

Name	Datatype	Description
application_type	string	Specifies whether the evaluation job is for evaluating a model or evaluating a knowledge base (retrieval and response generation). (ModelEvaluation, RagEvaluation)
creation_time	string (date-time)	The time the evaluation job was created.
customer_encryption_key_id	string	The Amazon Resource Name (ARN) of the customer managed encryption key specified when the evaluation job was created. (pattern: <code>(arn:aws(-[^:]+)?:kms:[a-zA-Z0-9-]*:[0-9]{12}:((key/[a-zA-Z0-9-]{36})|(alias/[a-zA-Z0-9-/]+)))|([a-zA-Z0-9-]{36})|(alias/[a-zA-Z0-9-/]+)</code>)
evaluation_config	object	The configuration details of either an automated or human-based evaluation job.
failure_messages	array	A list of strings that specify why the evaluation job failed to create.
inference_config	object	Contains the configuration details of the inference model used for the evaluation job.
job_arn	string	The Amazon Resource Name (ARN) of the evaluation job. (pattern: <code>arn:aws(-[^:]+)?:bedrock:[a-z0-9-]{1,20}:[0-9]{12}:evaluation-job/[a-z0-9]{12}</code>)
job_description	string	The description of the evaluation job. (pattern: <code>.+</code>)
job_name	string	The name for the evaluation job. (pattern: <code>[a-z0-9](-*[a-z0-9]){0,62}</code>)
job_type	string	Specifies whether the evaluation job is automated or human-based. (Human, Automated)
last_modified_time	string (date-time)	The time the evaluation job was last modified.
output_data_config	object	The Amazon S3 location where the results of your evaluation job are saved.
role_arn	string	The Amazon Resource Name (ARN) of the IAM service role used in the evaluation job. (pattern: <code>arn:aws(-[^:]+)?:iam::([0-9]{12})?:role/.+</code>)
status	string	The current status of the evaluation job. (InProgress, Completed, Failed, Stopping, Stopped, Deleting)

list_evaluation_jobs

Name	Datatype	Description
application_type	string	Specifies whether the evaluation job is for evaluating a model or evaluating a knowledge base (retrieval and response generation). (ModelEvaluation, RagEvaluation)
creation_time	string (date-time)	The time the evaluation job was created.
custom_metrics_evaluator_model_identifiers	array	The Amazon Resource Names (ARNs) of the models used to compute custom metrics in an Amazon Bedrock evaluation job.
evaluation_task_types	array	The type of task for model evaluation.
evaluator_model_identifiers	array	The Amazon Resource Names (ARNs) of the models used to compute the metrics for a knowledge base evaluation job.
inference_config_summary	object	Identifies the models, Knowledge Bases, or other RAG sources evaluated in a model or Knowledge Base evaluation job.
job_arn	string	The Amazon Resource Name (ARN) of the evaluation job. (pattern: <code>arn:aws(-[^:]+)?:bedrock:[a-z0-9-]{1,20}:[0-9]{12}:evaluation-job/[a-z0-9]{12}</code>)
job_name	string	The name for the evaluation job. (pattern: <code>[a-z0-9](-*[a-z0-9]){0,62}</code>)
job_type	string	Specifies whether the evaluation job is automated or human-based. (Human, Automated)
model_identifiers	array	The Amazon Resource Names (ARNs) of the model(s) used for the evaluation job.
rag_identifiers	array	The Amazon Resource Names (ARNs) of the knowledge base resources used for a knowledge base evaluation job.
status	string	The current status of the evaluation job. (InProgress, Completed, Failed, Stopping, Stopped, Deleting)

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_evaluation_job	select	job_identifier, region		Gets information about an evaluation job, such as the status of the job.
list_evaluation_jobs	select	region	creationTimeAfter, creationTimeBefore, statusEquals, applicationTypeEquals, nameContains, maxResults, nextToken, sortBy, sortOrder	Lists all existing evaluation jobs.
create_evaluation_job	insert	region, jobName, roleArn, evaluationConfig, inferenceConfig, outputDataConfig		Creates an evaluation job.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
job_identifier	string	The Amazon Resource Name (ARN) of the evaluation job you want get information on.
region	string	AWS region (default: us-east-1)
applicationTypeEquals	string	A filter to only list evaluation jobs that are either model evaluations or knowledge base evaluations.
creationTimeAfter	string (date-time)	A filter to only list evaluation jobs created after a specified time.
creationTimeBefore	string (date-time)	A filter to only list evaluation jobs created before a specified time.
maxResults	integer	The maximum number of results to return.
nameContains	string	A filter to only list evaluation jobs that contain a specified string in the job name.
nextToken	string	Continuation token from the previous response, for Amazon Bedrock to list the next set of results.
sortBy	string	Specifies a creation time to sort the list of evaluation jobs by when they were created.
sortOrder	string	Specifies whether to sort the list of evaluation jobs by either ascending or descending order.
statusEquals	string	A filter to only list evaluation jobs that are of a certain status.

SELECT examples

get_evaluation_job

Gets information about an evaluation job, such as the status of the job.

SELECT
application_type,
creation_time,
customer_encryption_key_id,
evaluation_config,
failure_messages,
inference_config,
job_arn,
job_description,
job_name,
job_type,
last_modified_time,
output_data_config,
role_arn,
status
FROM aws.bedrock.evaluation_jobs
WHERE job_identifier = '{{ job_identifier }}' -- required
AND region = '{{ region }}' -- required
;

list_evaluation_jobs

Lists all existing evaluation jobs.

SELECT
application_type,
creation_time,
custom_metrics_evaluator_model_identifiers,
evaluation_task_types,
evaluator_model_identifiers,
inference_config_summary,
job_arn,
job_name,
job_type,
model_identifiers,
rag_identifiers,
status
FROM aws.bedrock.evaluation_jobs
WHERE region = '{{ region }}' -- required
AND creationTimeAfter = '{{ creationTimeAfter }}'
AND creationTimeBefore = '{{ creationTimeBefore }}'
AND statusEquals = '{{ statusEquals }}'
AND applicationTypeEquals = '{{ applicationTypeEquals }}'
AND nameContains = '{{ nameContains }}'
AND maxResults = '{{ maxResults }}'
AND nextToken = '{{ nextToken }}'
AND sortBy = '{{ sortBy }}'
AND sortOrder = '{{ sortOrder }}'
;

INSERT examples

create_evaluation_job

Creates an evaluation job.

INSERT INTO aws.bedrock.evaluation_jobs (
jobName,
jobDescription,
clientRequestToken,
roleArn,
customerEncryptionKeyId,
jobTags,
applicationType,
evaluationConfig,
inferenceConfig,
outputDataConfig,
region
)
SELECT 
'{{ jobName }}' /* required */,
'{{ jobDescription }}',
'{{ clientRequestToken }}',
'{{ roleArn }}' /* required */,
'{{ customerEncryptionKeyId }}',
'{{ jobTags }}',
'{{ applicationType }}',
'{{ evaluationConfig }}' /* required */,
'{{ inferenceConfig }}' /* required */,
'{{ outputDataConfig }}' /* required */,
'{{ region }}'
RETURNING
job_arn
;

Manifest

# Description fields are for documentation purposes
- name: evaluation_jobs
props:
  - name: region
    value: "{{ region }}"
    description: Required parameter for the evaluation_jobs resource.
  - name: jobName
    value: "{{ jobName }}"
  - name: jobDescription
    value: "{{ jobDescription }}"
  - name: clientRequestToken
    value: "{{ clientRequestToken }}"
  - name: roleArn
    value: "{{ roleArn }}"
  - name: customerEncryptionKeyId
    value: "{{ customerEncryptionKeyId }}"
  - name: jobTags
    value:
      - key: "{{ key }}"
        value: "{{ value }}"
  - name: applicationType
    value: "{{ applicationType }}"
    valid_values: ['ModelEvaluation', 'RagEvaluation']
  - name: evaluationConfig
    description: |
      The configuration details of either an automated or human-based evaluation job.
    value:
      automated:
        datasetMetricConfigs:
          - taskType: "{{ taskType }}"
            dataset:
              name: "{{ name }}"
              datasetLocation:
                s3Uri: "{{ s3Uri }}"
            metricNames: "{{ metricNames }}"
        evaluatorModelConfig:
          bedrockEvaluatorModels:
            - modelIdentifier: "{{ modelIdentifier }}"
        customMetricConfig:
          customMetrics:
            - customMetricDefinition:
                name: "{{ name }}"
                instructions: "{{ instructions }}"
                ratingScale: "{{ ratingScale }}"
          evaluatorModelConfig:
            bedrockEvaluatorModels:
              - modelIdentifier: "{{ modelIdentifier }}"
      human:
        humanWorkflowConfig:
          flowDefinitionArn: "{{ flowDefinitionArn }}"
          instructions: "{{ instructions }}"
        customMetrics:
          - name: "{{ name }}"
            description: "{{ description }}"
            ratingMethod: "{{ ratingMethod }}"
        datasetMetricConfigs:
          - taskType: "{{ taskType }}"
            dataset:
              name: "{{ name }}"
              datasetLocation:
                s3Uri: "{{ s3Uri }}"
            metricNames: "{{ metricNames }}"
  - name: inferenceConfig
    description: |
      The configuration details of the inference model for an evaluation job. For automated model evaluation jobs, only a single model is supported. For human-based model evaluation jobs, your annotator can compare the responses for up to two different models.
    value:
      models:
        - bedrockModel:
            modelIdentifier: "{{ modelIdentifier }}"
            inferenceParams: "{{ inferenceParams }}"
            performanceConfig:
              latency: "{{ latency }}"
          precomputedInferenceSource:
            inferenceSourceIdentifier: "{{ inferenceSourceIdentifier }}"
      ragConfigs:
        - knowledgeBaseConfig:
            retrieveConfig:
              knowledgeBaseId: "{{ knowledgeBaseId }}"
              knowledgeBaseRetrievalConfiguration:
                vectorSearchConfiguration: "{{ vectorSearchConfiguration }}"
            retrieveAndGenerateConfig:
              type_: "{{ type_ }}"
              knowledgeBaseConfiguration:
                knowledgeBaseId: "{{ knowledgeBaseId }}"
                modelArn: "{{ modelArn }}"
                retrievalConfiguration: "{{ retrievalConfiguration }}"
                generationConfiguration: "{{ generationConfiguration }}"
                orchestrationConfiguration: "{{ orchestrationConfiguration }}"
              externalSourcesConfiguration:
                modelArn: "{{ modelArn }}"
                sources: "{{ sources }}"
                generationConfiguration: "{{ generationConfiguration }}"
          precomputedRagSourceConfig:
            retrieveSourceConfig:
              ragSourceIdentifier: "{{ ragSourceIdentifier }}"
            retrieveAndGenerateSourceConfig:
              ragSourceIdentifier: "{{ ragSourceIdentifier }}"
  - name: outputDataConfig
    description: |
      The Amazon S3 location where the results of your evaluation job are saved.
    value:
      s3Uri: "{{ s3Uri }}"