Set Up Evaluations

Add evaluations to a Flow, choose the right result type, map evaluator inputs, and decide when each evaluation should run.

6 min read

Evaluations turn an expected behavior into a repeatable check. Attach one or more evaluations to a Flow to measure requirements such as correctness, groundedness, response format, tone, or latency across individual requests and batch runs.

An evaluation is itself a Flow. It receives mapped data from the Flow being tested and returns a structured result that BotDojo can display in traces, batches, and analytics.

Add an evaluation to a Flow

Open the Flow you want to evaluate.
Select Evaluate.
Select Add Eval.
Choose an evaluator from Quick Setup, browse a category, or search for a template.

The evaluation template picker with Code Evaluation and LLM as a Judge options

The picker includes three ways to start:

Choice	Use it when
Starter	You want a simple Code Evaluation or LLM-as-a-Judge evaluator that you can name and customize.
Template	A reusable evaluator already covers the behavior you want to measure. Browse All or a category such as RAG.
Existing evaluation	An evaluator has already been added to this Project. Select it instead of creating another copy.

When a shared template is already present in the Project, Use Existing attaches that evaluator. Use the template action only when you intentionally want a separate editable copy.

Choose how the evaluation works

The evaluator's implementation and its result type answer different questions.

Evaluator implementation

Implementation	Best for	Tradeoffs
Code Evaluation	Exact values, schemas, required fields, patterns, length limits, and other deterministic rules	Fast and repeatable, but limited to rules you can express directly
LLM as a Judge	Correctness, relevance, tone, groundedness, completeness, and other qualities that require language understanding	Handles nuanced criteria, but adds model latency and cost and should be calibrated with examples
Custom evaluation Flow	Multi-step checks that combine code, models, tools, or organization-specific rules	Most flexible, but requires you to design and maintain the evaluator Flow

Start with a deterministic Code Evaluation when a rule can be checked exactly. Use an LLM as a Judge when a human would need to interpret the answer. Always inspect the judge's reason on representative examples before relying on its score.

Evaluation result types

Every evaluator declares one result type:

Result type	Returned value	Good examples
Metric	A number	Relevance from 0 to 1, quality from 1 to 5, or a similarity score
Pass/Fail	`true` or `false`	Valid schema, citation present, answer supported by context, or response under a limit
Classification	A category label	`relevant`, `off_topic`, `needs_review`, or a support-intent category

Choose Metric when you need a trend or want to compare versions. Choose Pass/Fail for a clear requirement. Choose Classification when the useful result is a named category rather than a numeric score.

An evaluator can also return a reason. Include one whenever possible so a failed or surprising result is actionable.

Map evaluator inputs

After choosing an evaluator, map each Eval Input to the value it should inspect.

The evaluation configuration showing evaluator inputs and Map from controls

Depending on the evaluator input type, Map from can include:

Individual Flow inputs or the complete Flow request body.
Individual Flow outputs or the complete Flow response.
The AI message content or complete AI message object.
Citations, session data, or the Flow trace.
A compatible output from a node in the Flow.
Map from Data Set for a reference answer, label, or other value that is supplied later by a batch.

For example, a correctness evaluator commonly maps:

Evaluator input	Map from
Question	The Flow's user-input field
Actual answer	The Flow output or AI message content
Expected answer	Map from Data Set

BotDojo only offers mappings with a compatible configured type. All required evaluator inputs must have a mapping before the evaluation can be saved.

Use Override Eval Properties only when this Flow needs evaluator settings that differ from the evaluator's defaults, such as a model, threshold, or instruction override.

Choose when the evaluation runs

Set Run Options before saving the evaluation.

The Run Options menu with request, session, thread, and manual choices

Run option	Use it when
Run automatically after each request	Each request can be judged independently. This is the usual choice for response-level checks.
Run automatically after each session	The evaluator needs the completed conversation or other session-level context.
Run automatically after each thread	The evaluator should judge one completed thread within a longer session.
Run manually	The evaluation is expensive, needs dataset-only inputs, or should run only from a trace, review, or batch workflow.

Automatic LLM-as-a-Judge evaluations add model usage and can increase the work performed for each request. Use session, thread, or manual execution when the evaluation does not need to run after every response.

Select Save when the mappings and run option are correct.

Test and inspect the result

Run the Flow with a representative input.
Open Trace and select the evaluation result.
Check the returned value and reason, not only whether it appears successful.
Test edge cases that should pass and fail. Refine the evaluator prompt, deterministic rule, mapping, or threshold when the result does not match the intended requirement.

For repeatable testing, add the evaluation to a batch and map dataset columns to any inputs marked Map from Data Set. Keep the dataset stable when comparing model, prompt, retrieval, or Flow changes.

Continue with Run Evaluations, Custom Evaluations, or Create and Run a Batch.

Troubleshooting

A required input cannot be saved

Confirm that every required evaluator input has a mapping. If the expected value comes from a dataset, choose Map from Data Set now and complete the dataset-column mapping when creating the batch.

The value you need is not available in Map from

Mapping options are filtered by type. Confirm that the Flow input, output, or node output uses the same configured type as the evaluator input.

The evaluation did not run

Open the evaluation configuration and check Run Options. A manual evaluation does not run automatically with every Flow request. Also confirm that batch runs include the evaluation and all dataset-supplied inputs.

An LLM judge is inconsistent

Make the criteria explicit, include examples of passing and failing responses, and test against a fixed dataset. Use a deterministic Code Evaluation instead when the requirement can be expressed as an exact rule.