> ## Documentation Index
> Fetch the complete documentation index at: https://docs.budecosystem.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluating Agents

> Measure a published agent end to end: its answers, the tools it called, and how it behaves over a conversation

An agent evaluation runs your test cases **through the agent itself** — the same `/v1/responses` call
your users make, with the agent's tools, knowledge and policies — and scores what came back. Every
sample keeps its full trajectory, so a score can always be traced to the steps and tool calls that
produced it.

> [!WARNING]
>
> Every agent call in an evaluation is **real**. If the agent can create tickets, send messages or
> change data through its tools, an evaluation will do so. The run plan lists every tool that may have
> side effects and asks you to acknowledge them before the run starts.

## What is measured

| Kind | Metrics | Judge model |
| - | - | - |
| Answer quality | answer relevancy, faithfulness, hallucination, G-Eval, task completion and the other catalog metrics | yes |
| Tool use | tool correctness (expected tools vs tools called), argument correctness, MCP use | tool correctness: no |
| Conversations | conversation completeness, turn relevancy, knowledge retention, role adherence | yes |
| Trajectory (always reported) | success rate, steps, tool calls, tool error rate, latency p50 / p95, tokens, estimated cost | no |
| Repeats | pass^k — the share of test cases that pass on **every** one of *k* runs | — |

Tool correctness compares `expected_tools` with what the agent called. An explicit empty list
(`"expected_tools": []`) means "no tool should be called" and is scored; leaving the key out skips the
metric for that test case.

## Public benchmarks

Hub benchmarks (ARC, MMLU, GSM8K and the rest) also run against an agent. Each benchmark question is
sent to the agent as a normal `/v1/responses` call, and the benchmark's own answer extraction and
scoring grade the reply, so no judge model is needed.

* The score measures the **agent** (its instructions, tools and policies), not the model behind it.
  It is labelled *not comparable* with model benchmark scores.
* Only **generation** configs run. A benchmark scored by log-probabilities (`ppl` / `ll` configs)
  needs token probabilities an agent does not return; the generation config is used when the
  benchmark has one, and the run is refused when it has none.
* Every question is a real agent call, so the run plan caps the items per benchmark (200 by default,
  **Max items per benchmark**). The call and cost caps and the failure breaker of a test-case run do
  not apply to benchmarks; the item cap is the bound.
* A call the agent does not finish (paused for an approval, denied by a policy, cut off) scores as
  no answer.
* Benchmarks and custom or red-team evaluations run as **separate** evaluations on an agent.

## Red team

A red-team evaluation attacks the agent with the vulnerabilities and attack methods you pick, through
the same `/v1/responses` call. The attacker model defaults to the judge model when you do not pick
one. Each finding keeps the agent's response id and trace, so it can be opened in the agent's runs.

## Test cases

* **Single turn** — `input` (or `input_items` with up to four images), optional `expected_output`,
  `expected_tools`, `context`, `variables`.
* **Scripted conversation** — `turns`: the user messages, sent one after another in one conversation.
* **Simulated conversation** — a `scenario`, `expected_outcome` and optional `user_description`; a
  simulated user (the judge model unless you pick another) talks to the agent for up to `max_turns`.
  Each sample records whether the simulated user stayed on its goal.

## Running an evaluation

1. Open an experiment and choose **Run evaluation**.
2. Pick what to run (Hub benchmarks, custom evaluations or red team), then which ones, and name the run.
3. Under **What do you want to evaluate?** pick **Agent**, then the agent. An agent with more than one
   version asks which version to test, with the default selected.
4. Pick a judge model. This step appears only for custom evaluations and red team; a benchmark is
   scored by its own evaluator and needs no judge.
5. Review the **run plan**: the planned calls (judge and simulated-user calls appear only when the run
   makes them), the estimated cost range and any warnings. Starting the run acknowledges that the
   agent's tools are called for real.
6. Optional settings: concurrency, repeats, a cost or call cap, the failure threshold, whether test
   case text is kept after scoring.

### Identity

MCP connector tools run as the **platform admin**, not as the person who started the run. Red-team
findings about authorization (BFLA, RBAC, BOLA) are labelled with that identity.

### Cost and rate limits

Costs are **estimates** from token usage and each model's own price (the catalogue price of the agent's
model and of the judge model, else the price set on the deployment),
shown as "estimated" and never mixed with billing. A component without a price shows as *unpriced*,
never as zero. Evaluation calls count against the agent's and the deployment's rate limits like any
other traffic; evaluation traffic is excluded from production analytics by default (switch **Include
evaluation runs** on the agent's overview to see it) and appears as its own line in usage.

## Outcomes

| Sample status | Meaning |
| - | - |
| completed | the agent answered; scored |
| requires\_action | the agent stopped for an approval; can be scored later with **Resume gated** |
| denied / incomplete | a governance policy blocked or cut the answer (the policy is named) |
| failed / timeout / error | the call failed; not scored, can be re-run |
| cancelled / interrupted | never started because the run was stopped, or cut by a restart |

A run is **completed**, **partial** (for example the judge became unavailable — re-score it later),
**cancelled** or **failed** (for example the agent was unreachable for several samples in a row, which
stops the run early instead of failing every remaining sample).

## While and after a run

* **Cancel** drains: calls already in flight finish and are kept; samples not yet started are recorded
  as cancelled. Partial aggregates are kept.
* **Re-score** scores the stored trajectories again with a different judge, metric set or threshold —
  the agent is **not** called again. Previous scores are kept as history.
* **Re-run failed** retries only failed, timed-out, errored, interrupted or cancelled samples.
* **Review** lets a person mark samples pass / fail; the summary shows the human pass rate and how often
  the judge agreed.
* **Score past runs** (agent page → Evaluations) scores stored production runs of the agent without
  calling it. Evaluation traffic is never selected.

## Automating in CI

Trigger an evaluation with a **project API key** and read the gate:

```bash theme={null}
curl -X POST "$BUD_APP_URL/experiments/$EXPERIMENT_ID/evaluations/trigger" \
  -H "Authorization: Bearer $BUD_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "nightly", "target": {"target_type": "agent", "prompt_id": "'$AGENT_ID'"},
       "datasets": [{"dataset_id": "'$DATASET_ID'"}], "judge_endpoint_id": "'$JUDGE_ID'",
       "eval_options": {"acknowledge_side_effects": true,
                        "gate": {"answer_relevancy": {"min": 0.8}, "agent_success_rate": {"min": 0.9}},
                        "callback": {"url": "https://ci.example.com/hooks/bud"}}}'
```

Add `?plan=true` to get the plan without starting anything. The key must belong to the experiment's
project. Poll `GET /experiments/evaluations/{evaluation_id}/gate`, or receive the signed webhook:
create the project's webhook secret first (`POST /experiments/webhook-secret` with the
`project_id`; it is shown once), then verify each delivery. A callback and a gate are set through
this API only; the run wizard does not ask for them.

```python theme={null}
import hashlib, hmac

def verify(secret: str, raw_body: bytes, header: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header)  # header: X-Bud-Signature
```

Callback URLs must be `https` and resolve to a public address. Deliveries are retried three times and
logged on the evaluation; a failed delivery never fails the evaluation.

## Retention and deletion

Sample rows are kept for 180 days by default, then removed; the samples view then says when they
expired. Deleting an experiment with **purge** removes its stored samples immediately.

## Limitations

* Runs of agent versions saved before agent attribution existed cannot be scored as past runs; save
  the version again to attribute its new runs.
* Sub-agent trajectories are attached only for delegations that paused (for example for an approval).
* DAG, conversational DAG, arena G-Eval and the image metrics are not scored on an agent (they need a
  decision graph, competing outputs or images); they show as skipped. Retrieval metrics (contextual
  precision / recall / relevancy, faithfulness) are scored only when the agent retrieved from its
  knowledge for that sample.
* While an evaluation service instance restarts, its in-flight samples finish first; a retried run
  continues from the samples already stored.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.