BotDojoDocs
Search BotDojo Docs
Search every guide

Try “stop a run,” “connect an app,” or “daily briefing.”

Browse Platform
PlatformProjectCompare Batch Runs
Platform · Project

Compare Batch Runs

Compare completed BotDojo batch runs by quality, cost, latency, and individual request results before choosing a model, prompt, or flow configuration.

4 min read

Use a batch comparison to understand the tradeoffs between two or more versions of an AI application. Start with the summary to compare quality, cost, and speed. Then inspect matching requests to understand why the results differ.

Before you compare

Choose completed runs that isolate the change you want to evaluate. For the clearest result:

  • Use the same Flow and dataset.
  • Keep the dataset mappings and evaluations the same.
  • Change one variable at a time, such as the model, prompt, tool, or retrieval configuration.
  • Wait for every run to finish before comparing it.

If several variables changed, the comparison still shows what happened, but it may not tell you which change caused the difference.

Select batch runs

  1. Open the Flow and select Batch.
  2. Select the completed runs you want to compare.
  3. Confirm that the selected rows use the expected Flow and Data Set.
  4. Select Compare Batches.
Two completed batch runs selected in the Batch table, with Compare Batches available above the table.
Screenshot previewTwo completed batch runs selected in the Batch table, with Compare Batches available above the table.
Two completed batch runs selected in the Batch table, with Compare Batches available above the table.

In this example, MiniMax M3 and GLM 5.2 both ran the same 23-item dataset through the same Flow. The model is the variable under test.

Read the Summary

The Summary tab presents aggregate measurements for each selected batch:

  • Total Cost — the tracked model and tool cost for the run.
  • Avg Request Duration — the average time required to process one dataset item.
  • Total Tokens — the combined token usage across the run.
  • Total Error — requests that finished with an error.
  • Total Calls — model and tool calls made while processing the batch.
  • Percentage Passed — the percentage of evaluation checks that passed.
Compare Batches Summary showing cost, duration, token, error, call, and evaluation pass measurements for GLM 5.2 and MiniMax M3.
Screenshot previewCompare Batches Summary showing cost, duration, token, error, call, and evaluation pass measurements for GLM 5.2 and MiniMax M3.
Compare Batches Summary showing cost, duration, token, error, call, and evaluation pass measurements for GLM 5.2 and MiniMax M3.

For the example runs:

MeasurementGLM 5.2MiniMax M3
Total Cost$2.082374$0.453839
Avg Request Duration16.961s7.137s
Total Tokens1,277,1851,224,888
Total Error00
Total Calls343329
Percentage Passed91.84%91.49%

MiniMax M3 cost about 78% less and averaged about 58% less time per request, while its pass rate was 0.35 percentage points lower. That is a useful signal, but it is not enough to choose a winner. Open the request data to find where the quality difference occurred and whether it matters for your application.

Comparison note: Percentage Passed is comparable only when the runs use the same evaluation configuration. A similar aggregate pass rate can also hide important differences on individual dataset items.

Inspect matching results

Select the Data tab to inspect request-level results. BotDojo groups requests by Dataset Item ID, which keeps the results for the same input together.

  1. Expand a dataset item to see one request from each selected batch.
  2. Review the batch name, duration, status, input, output, and evaluation columns.
  3. Use filters or Show/Hide columns to focus on failures or the evaluations that matter to your test.
  4. Select the matching request rows and select Compare Requests.
The Data tab in Compare Batches with one Dataset Item ID expanded to show the GLM 5.2 and MiniMax M3 requests for the same question.
Screenshot previewThe Data tab in Compare Batches with one Dataset Item ID expanded to show the GLM 5.2 and MiniMax M3 requests for the same question.
The Data tab in Compare Batches with one Dataset Item ID expanded to show the GLM 5.2 and MiniMax M3 requests for the same question.

The request comparison places the matching inputs and outputs side by side. Continue down the comparison to review each evaluation result and score. This is where you can distinguish a harmless wording difference from a regression in correctness, groundedness, citations, or another evaluation used by your Flow.

Compare Requests showing the same input and the two generated outputs side by side.
Screenshot previewCompare Requests showing the same input and the two generated outputs side by side.
Compare Requests showing the same input and the two generated outputs side by side.

Choose the run to keep

Define the quality requirement first, then use cost and latency to choose among the runs that meet it. A practical review sequence is:

  1. Check errors and the overall evaluation pass rate.
  2. Inspect failed or changed dataset items.
  3. Confirm that important inputs still produce acceptable outputs.
  4. Compare cost, duration, tokens, and calls.
  5. Repeat the experiment if the result is close or the changed sample is too small.

For example, you might accept the faster, lower-cost model if its only changed result is an acceptable wording variation. Keep the higher-quality model if the cheaper run fails a critical policy, correctness, or groundedness check.

Next steps