Generating and Running Evals
OmniAgent::Eval lets you test agent quality, not just correctness — deterministic assertions (tool calls, output matching) plus pluggable LLM-as-judge scoring.
Golden Sets
For many input/expected-output pairs, load a dataset file instead of writing a eval_case per row:
Judge Provider
judge resolves which provider grades the output, in this order:
Caching
Eval runs are cached by default, keyed on (agent class, runalias, input, context), and stored as JSON at OmniAgent.configuration.evalcachepath (default tmp/omniagentevalcache.json). Re-running the same case — e.g. while iterating on assertions — replays the cached output instead of calling the provider again, so you don't spend tokens testing the same case twice.