Judge Provider
judge resolves which provider grades the output, in this order:
-
An explicit
provider:(and optionalmodel:) kwarg passed tojudge. -
OmniAgent.configuration.eval_judge_provider/eval_judge_model, set once globally:OmniAgent.configure do |config|config.eval_judge_provider = :openaiconfig.eval_judge_model = "gpt-4o-mini"end -
Falls back to the same provider/model the agent under test already uses — no judge setup required to get started.
The judge sends a fixed grading prompt asking for {"score": 0-1, "reason": "..."} and compares score against threshold from the judge call (see Generating and Running Evals).