Judge panel

The judge panel is the eval. It reviews a turn's answer on its own.

Goal enforcement checks that the loop keeps moving; it says nothing about whether the final answer is actually right. The panel is an independent review of that answer, run after the turn produces it - one or more reviewer personas, each with real tool access, check the output and can send it back for revision before it reaches you.

text

answer -> [judge 1] -> [judge 2] -> [judge 3] -> all approved? -> answer
              |             |             |
              v (tools)     v (tools)     v (tools)
          read files    run tests    search the web
                    any rejected -> revise -> re-judge

A judge is a full agent, not a rubber stamp. Each judge runs its own ephemeral tool-calling loop (the same round loop the main turn uses), so it can read a file, run a test, or search the web to verify a claim before ruling - not just re-read the answer and guess. A judge settles its review by calling judge_verdict{approved, feedback} exactly once; there is no way to approve or reject by prose alone.

The panel, not one judge. Every judge in the roster reviews the same output, concurrently. If any judge rejects it, the rejecting judges' feedback is combined into a revision directive, the loop produces a new answer, and the whole panel reviews it again - up to JudgeMaxIterations times (default 2). Once the budget is spent the last attempted answer is returned regardless - a judge panel that never approves must not be able to block the turn forever. A revision that comes back empty (or with the "model produced nothing" notice) is discarded and the previous answer is kept, so a failed revision never replaces a real answer with a blank or a canned message.

You can see the roster as soon as it is generated. Auto-judges run one LLM call at the start of the turn (the same synthetic JSON call the loop uses for compaction). The REPL prints the panel before the main tool loop, so you know who will review before any work starts:

[judges] panel generated - 2 reviewer(s)
  - correctness: every factual claim must be verifiable from the codebase
  - security: no secrets, credentials, or unsafe shell input

If the roster call comes back empty - the model is unavailable, or its reply would not parse - the REPL says so rather than silently skipping review:

[judges] no panel this turn - roster generation returned nothing

/judges (or /judges list) shows that last auto-generated panel. An explicit roster from /judges add still wins at review time and is not re-printed every turn.

You can see the review happening. Because a judge is a full tool-calling loop, a review can take as long as a real turn - the REPL prints [judge] status lines so you are never left staring at a silent prompt:

[judge] reviewing output - correctness, security
[judge] approved by correctness, security

or, on rejection:

[judge] reviewing output - correctness, security
[judge] revision requested (1/2): security: uses a shell command with unescaped input
[judge] reviewing output - correctness, security
[judge] approved by correctness, security

Configuring the roster.

cfg := agent.Config{
    // Explicit roster - always wins when set.
    Judges: []agent.JudgeSpec{
        {Name: "correctness", Criteria: "every factual claim must be verifiable from the codebase"},
        {Name: "security", Criteria: "no secrets, credentials, or unsafe shell input"},
    },
    // Or: generate a roster (1-3 judges) per turn instead of hand-configuring one.
    AutoJudges: true,
    JudgeMaxIterations: 2, // revise-and-rejudge budget
}

A JudgeSpec.Tools list scopes which registered tools that judge may call (empty means every tool available to the main loop). With AutoJudges, the roster is generated by one LLM call per turn - the same synthetic-workflow JSON-mode pattern compaction uses - and skipped for trivial chat the same way compaction skips it. Any failure anywhere in the panel (a judge that errors, times out, or never calls judge_verdict) degrades to approval rather than blocking the turn.

On unless turned off. The REPL and the desktop agent loop start with the panel on. A settings file that never set auto_judges stays on. /judges off (or /judges clear) drops the roster and writes auto_judges: false. /judges auto on turns it back on. A hand-configured roster (/judges add / remove) is session-only and wins over the generated one. Library callers get no panel unless they set Config.Judges or Config.AutoJudges. When the agent calls a workflow or an agency as a tool, the panel reviews that turn's final answer. An agency started as its own agent loop gets the same default. kdeps run does not.

Graders sit on a judge and run on the answer first, in process, with no model call. A failed grader rejects the answer and that judge does not open an LLM review. Graders that pass, on a judge that also has criteria, still use judge_verdict. A judge with graders and no criteria never calls the model. The auto-generated roster stays criteria-only. Kinds: exact, contains, not_contains, regex, json (valid JSON, or equal to a JSON pattern), expr (output and answer are the candidate text). /judges prints them under that judge. /judges off drops them with the roster.

From the REPL, the same roster is managed with /judges - no restart needed:

/judges                                        # show the current roster / auto-judges state
/judges auto off                               # turn the panel off and store it
/judges add correctness every claim must cite a source   # or: hand-configure an explicit roster instead
/judges grade correctness contains workflow.yaml         # attach a grader to that judge
/judges remove correctness                     # drop it again
/judges clear                                  # disable the panel entirely

Judge panel ​

See also ​

Judge panel

See also