Evaluating Agents

​

What is measured

​

Public benchmarks

• The score measures the agent (its instructions, tools and policies), not the model behind it. It is labelled not comparable with model benchmark scores.
• Only generation configs run. A benchmark scored by log-probabilities (ppl / ll configs) needs token probabilities an agent does not return; the generation config is used when the benchmark has one, and the run is refused when it has none.
• Every question is a real agent call, so the run plan caps the items per benchmark (200 by default, Max items per benchmark). The call and cost caps and the failure breaker of a test-case run do not apply to benchmarks; the item cap is the bound.
• A call the agent does not finish (paused for an approval, denied by a policy, cut off) scores as no answer.
• Benchmarks and custom or red-team evaluations run as separate evaluations on an agent.

​

Red team

​

Test cases

• Single turn — input (or input_items with up to four images), optional expected_output, expected_tools, context, variables.
• Scripted conversation — turns: the user messages, sent one after another in one conversation.
• Simulated conversation — a scenario, expected_outcome and optional user_description; a simulated user (the judge model unless you pick another) talks to the agent for up to max_turns. Each sample records whether the simulated user stayed on its goal.

​

Running an evaluation

1. Open an experiment and choose Run evaluation.
2. Pick what to run (Hub benchmarks, custom evaluations or red team), then which ones, and name the run.
3. Under What do you want to evaluate? pick Agent, then the agent. An agent with more than one version asks which version to test, with the default selected.
4. Pick a judge model. This step appears only for custom evaluations and red team; a benchmark is scored by its own evaluator and needs no judge.
5. Review the run plan: the planned calls (judge and simulated-user calls appear only when the run makes them), the estimated cost range and any warnings. Starting the run acknowledges that the agent’s tools are called for real.
6. Optional settings: concurrency, repeats, a cost or call cap, the failure threshold, whether test case text is kept after scoring.

​

Identity

​

Cost and rate limits

​

Outcomes

​

While and after a run

• Cancel drains: calls already in flight finish and are kept; samples not yet started are recorded as cancelled. Partial aggregates are kept.
• Re-score scores the stored trajectories again with a different judge, metric set or threshold — the agent is not called again. Previous scores are kept as history.
• Re-run failed retries only failed, timed-out, errored, interrupted or cancelled samples.
• Review lets a person mark samples pass / fail; the summary shows the human pass rate and how often the judge agreed.
• Score past runs (agent page → Evaluations) scores stored production runs of the agent without calling it. Evaluation traffic is never selected.

​

Automating in CI

curl -X POST "$BUD_APP_URL/experiments/$EXPERIMENT_ID/evaluations/trigger" \
  -H "Authorization: Bearer $BUD_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "nightly", "target": {"target_type": "agent", "prompt_id": "'$AGENT_ID'"},
       "datasets": [{"dataset_id": "'$DATASET_ID'"}], "judge_endpoint_id": "'$JUDGE_ID'",
       "eval_options": {"acknowledge_side_effects": true,
                        "gate": {"answer_relevancy": {"min": 0.8}, "agent_success_rate": {"min": 0.9}},
                        "callback": {"url": "https://ci.example.com/hooks/bud"}}}'

import hashlib, hmac

def verify(secret: str, raw_body: bytes, header: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header)  # header: X-Bud-Signature

​

Retention and deletion

​

Limitations

• Runs of agent versions saved before agent attribution existed cannot be scored as past runs; save the version again to attribute its new runs.
• Sub-agent trajectories are attached only for delegations that paused (for example for an approval).
• DAG, conversational DAG, arena G-Eval and the image metrics are not scored on an agent (they need a decision graph, competing outputs or images); they show as skipped. Retrieval metrics (contextual precision / recall / relevancy, faithfulness) are scored only when the agent retrieved from its knowledge for that sample.
• While an evaluation service instance restarts, its in-flight samples finish first; a retried run continues from the samples already stored.