Evaluations help you check whether an agent behaves correctly across repeated test cases. Use evaluations when you need evidence that an agent change improved or preserved behavior.

Evaluation routes

Frontend route:
Backend routes:

Evaluation workflow

1

Create an evaluation set

Group related test cases under one evaluation set.
2

Add cases

Add prompts and expected criteria for the target agent behavior.
3

Run the evaluation

Execute the evaluation set against an agent.
4

Review the run

Inspect run output and compare results over time.

When to evaluate

  • Before publishing an agent.
  • After changing tools or model configuration.
  • After changing the agent instruction.
  • After backend execution changes.

Evaluation run features

Incremental streaming results

During an evaluation run, Pinter streams test case outcomes incrementally:
  • Case passes and failures are rendered in real time as each case finishes, without waiting for the entire batch to finish.
  • Intermediate scores and tool trajectory matches update live on the dashboard.

Agent version binding and historical audit

Evaluation runs record the exact version number of the target agent (agent_version). This ensures that score history reflects whether a regression happened due to a prompt change, a tool change, or a model switch.
Keep evaluation cases small and tied to real user questions. Broad prompts are harder to diagnose when they fail.