LLM-as-a-Judge In Spring AI: A Practical Guide

LLM-as-a-Judge in Spring AI: A Practical Guide

Imagine you have just built a Spring AI application that answers customer questions from your company’s knowledge base. The application works beautifully in your local environment. You ask, “What is your refund policy?” and the AI gives a clear, confident answer.

But then comes the uncomfortable question every developer eventually asks:

How do you know the answer is actually good?

Traditional software gives us familiar ways to test this. We can assert that a method returns the expected value, check whether an API returns HTTP 200, or verify that a database contains the expected record. Generative AI is different. The same question can produce different answers, and a response can sound perfectly convincing while being incomplete, irrelevant, or factually wrong.

You could ask a human to review every response, but that quickly becomes expensive and difficult to scale. This is where the idea of LLM-as-a-Judge becomes interesting. Instead of asking only a human to evaluate an AI response, we can use another LLM to assess the response against specific criteria such as relevance, factual accuracy, faithfulness to the supplied context, and instruction adherence. Spring AI provides evaluation abstractions such as Evaluator, along with evaluators such as RelevancyEvaluator and FactCheckingEvaluator, to support this kind of automated assessment.

Evaluation testing catches a bad response before you ship. LLM-as-a-Judge catches it after the model generates it, live, and fixes it before the user ever sees it. Spring AI implements this through Recursive Advisors, a CallAdvisor that scores its own output and retries with feedback until it clears a quality bar. Here’s how it actually works, and when it’s worth the extra LLM calls.

Table of Contents

What Is LLM-as-a-Judge?

LLM-as-a-Judge is a technique where one language model scores the output of another, using criteria like relevance, accuracy, or completeness, instead of relying only on human review or brittle string-matching metrics. In Spring AI, it’s implemented through Recursive Advisors, which let a ChatClient call chain loop back and retry a weak response automatically.

It works because judging is an easier task than generating. Spotting a missing detail in a summary takes far less reasoning than writing a complete summary from scratch, the same reason it’s easier to critique an essay than to write one. Research cited in Spring AI’s own guide puts sophisticated judge models at roughly 85% agreement with human raters, ahead of typical human-to-human agreement of 81%.

How Does LLM-as-a-Judge in Spring AI Work?

Spring AI’s Recursive Advisors extend the normal CallAdvisor interface with the ability to call back into the same advisor chain. Instead of a linear request-response-response flow, the advisor can inspect the response, decide it’s not good enough, mutate the request with feedback, and call the chain again.

There are two established evaluation patterns behind this:

Direct Assessment (point-wise scoring): the judge rates a single response on a scale, commonly 1 to 4, and that score drives a retry decision. This is what most Spring AI examples build.
Pairwise Comparison: the judge picks the better of two candidate responses, useful for A/B testing prompt or model changes rather than gating a single live call.

Spring AI’s official guide walks through Direct Assessment via a SelfRefineEvaluationAdvisor. Below, we build a variant of that same idea against a different problem so you can see how the pattern generalizes.

LLM-as-a-Judge vs Spring AI Evaluation Testing: What’s the Difference?

These two get conflated constantly, and they solve different problems. If you’ve read our Spring AI Evaluation Testing guide, you already know RelevancyEvaluator and FactCheckingEvaluator, both Evaluator implementations meant for JUnit tests, run before deploy.

LLM-as-a-Judge, via Recursive Advisors, runs the same underlying idea (a model judging a model) at request time, inside the live call chain, in production. It doesn’t replace your test suite, it adds a runtime safety net for the failures your tests didn’t catch.

	Evaluation Testing (Evaluator)	LLM-as-a-Judge (Recursive Advisors)
When it runs	Build time, in JUnit	Runtime, on every live call
Purpose	Catch regressions before deploy	Catch and fix a bad response before the user sees it
Cost model	Fixed, part of CI	Variable, scales with retries per request
Spring AI status	Stable since 1.0	Experimental since 1.1.0-M4

In a production setup, you generally want both: evaluator tests as your CI gate, and a judge advisor reserved for the highest-stakes live endpoints where a retry is worth the extra latency.

Prerequisites

Comfortable with Spring AI’s ChatClient and Advisors API
Spring Boot 4.1 with Spring AI 2.0 (Recursive Advisors work from 1.1.0-M4 onward, so 1.1.x also qualifies if you’re not yet on 2.0)
Ideally, you’ve already read the Evaluation Testing guide, since this article assumes you know what EvaluationRequest/EvaluationResponse do
A second, cheaper chat model available for judging (a local Ollama model works fine)

Step#1: Add the Dependencies

Recursive Advisors ship inside Spring AI’s core ChatClient module, so no separate starter is needed beyond whatever chat model starters you’re already using.

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>2.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-openai</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-ollama</artifactId>
    </dependency>
</dependencies>

OpenAI (or your production model) generates; a local Ollama model judges, keeping the judge call cheap regardless of how many retries happen.

Step#2: Build a Judge-Based Retry Advisor

Say you’re generating meeting summaries from a transcript, and the failure mode you care about is a summary that drops key decisions or action items. Here’s a CallAdvisor that judges completeness and retries with feedback:

public class MeetingSummaryJudgeAdvisor implements CallAdvisor {

    private static final PromptTemplate JUDGE_PROMPT = new PromptTemplate("""
            Rate how completely the summary captures the key decisions and
            action items from the transcript, on a scale of 1 to 4.
            1 = missing most decisions and action items
            4 = captures all decisions and action items clearly

            Transcript: {transcript}
            Summary: {summary}

            Respond as JSON matching: {"rating": <1-4>, "feedback": "<what is missing>"}
            """);

    record JudgeVerdict(int rating, String feedback) {}

    private final ChatClient judgeClient;
    private final int minRating;
    private final int maxAttempts;

    public MeetingSummaryJudgeAdvisor(ChatClient.Builder judgeBuilder, int minRating, int maxAttempts) {
        this.judgeClient = judgeBuilder.build();
        this.minRating = minRating;
        this.maxAttempts = maxAttempts;
    }

    @Override
    public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
        ChatClientRequest currentRequest = request;

        for (int attempt = 1; attempt <= maxAttempts; attempt++) {
            ChatClientResponse response = chain.copy(this).nextCall(currentRequest);
            JudgeVerdict verdict = judge(currentRequest, response);

            if (verdict.rating() >= minRating || attempt == maxAttempts) {
                return response;
            }

            currentRequest = withFeedback(currentRequest, verdict);
        }

        throw new IllegalStateException("Unreachable: loop always returns or exhausts attempts");
    }

    private JudgeVerdict judge(ChatClientRequest request, ChatClientResponse response) {
        String prompt = JUDGE_PROMPT.render(Map.of(
                "transcript", request.prompt().getUserMessage().getText(),
                "summary", response.chatResponse().getResult().getOutput().getText()
        ));
        return judgeClient.prompt(prompt).call().entity(JudgeVerdict.class);
    }

    private ChatClientRequest withFeedback(ChatClientRequest request, JudgeVerdict verdict) {
        Prompt augmented = request.prompt().augmentUserMessage(msg -> msg.mutate()
                .text(msg.getText() + "\n\nPrevious summary rated " + verdict.rating()
                        + "/4. Missing: " + verdict.feedback())
                .build());
        return request.mutate().prompt(augmented).build();
    }
}

The chain.copy(this).nextCall(currentRequest) call is what makes this recursive: it re-enters the advisor chain rather than returning immediately, letting the loop try again with the augmented prompt.

Step#3: Wire the Advisor into a ChatClient

@Bean
ChatClient summaryChatClient(OpenAiChatModel openAiChatModel, OllamaChatModel ollamaChatModel) {
    return ChatClient.builder(openAiChatModel)
            .defaultAdvisors(
                new MeetingSummaryJudgeAdvisor(ChatClient.builder(ollamaChatModel), 3, 4)
            )
            .build();
}

This requires a minimum rating of 3 out of 4, with up to 4 attempts before giving up and returning the last response anyway. Every failed attempt costs one extra generation call plus one judge call, so a summary that struggles to satisfy the judge can burn through several LLM calls before it succeeds or the attempts run out.

Step#4: Choose a Judge Model and Mitigate Bias

Using the same model instance to generate and judge risks narcissistic bias: a model tends to rate its own phrasing more favorably than an independent model would. Spring AI’s guide addresses this by using a separate ChatClient for evaluation, exactly as Step#3 does with a distinct Ollama-backed client for judging.

A few things worth locking down on the judge specifically:

Temperature at or near 0. A judge with high temperature makes your retry logic non-deterministic on top of the generation model already being non-deterministic.
A dedicated judge model where possible. The Judge Arena Leaderboard tracks models specifically benchmarked for judging tasks, which consistently outperform general-purpose models used as judges.
Structured output for the verdict. Parsing the rating into a typed record, as JudgeVerdict does above, avoids brittle text-scraping for the score.

When Should You Use LLM-as-a-Judge in Production?

Not every endpoint needs a judge in the loop. It’s worth it when a wrong or incomplete response is expensive to get wrong, think compliance summaries, customer-facing policy answers, or anything where a retry is cheaper than the cost of the mistake reaching a user. It’s a poor fit for high-throughput, low-stakes endpoints where doubling or tripling the LLM calls per request isn’t worth the latency and cost for a minor quality gain.

Two hard constraints matter here too. Recursive Advisors are non-streaming only as of Spring AI 2.0, so this pattern doesn’t currently work with .stream() calls. And because the advisor calls back into the chain, advisor ordering matters more than usual, an advisor with external side effects (like writing to a database) placed before the judge advisor in the chain can fire multiple times per request if you’re not careful.

Common Mistakes and Confusion Points

Treating this as a replacement for evaluation testing. It’s a runtime safety net, not a substitute for the JUnit-level RelevancyEvaluator/FactCheckingEvaluator checks that should already be catching regressions in CI.
No termination condition. Without a hard maxAttempts cap, a judge that never approves a response creates an effectively infinite retry loop. Always cap it, and always handle the case where the cap is hit.
Applying it to streaming endpoints. The current implementation doesn’t support .stream(), so wrapping a streaming chat endpoint with a judge advisor will not behave as expected.
Ignoring advisor order. Advisors with side effects (logging to an external system, decrementing a rate limit, writing to a database) need careful placement relative to the judge advisor, since a retry re-runs the chain.
Underestimating cost at scale. Each failed attempt is two extra LLM calls. A judge advisor set to 4 attempts can, in the worst case, quadruple your per-request LLM spend on that endpoint.

Interview Questions on This Topic

Q#1. What is LLM-as-a-Judge, and why use it over metrics like ROUGE or BLEU?

LLM-as-a-Judge uses a language model to assess quality dimensions like relevance and completeness, which correlates better with human judgment than n-gram overlap metrics like ROUGE or BLEU, especially for open-ended, nuanced responses.

Q#2. How does LLM-as-a-Judge differ from Spring AI’s Evaluator-based testing?

Evaluator implementations like RelevancyEvaluator run in JUnit at build time to catch regressions before deploy. LLM-as-a-Judge, via Recursive Advisors, runs live in the request path and can automatically retry a weak response before the user sees it.

Q#3. What is a Recursive Advisor, and what makes it “recursive”?

It’s a CallAdvisor that can call back into its own advisor chain using chain.copy(this).nextCall(…), letting it loop, evaluate, and retry with an augmented request instead of returning after a single pass.

Q#4. Why does Recursive Advisors’ non-streaming limitation matter for production systems?

Many chat UIs rely on streaming for responsiveness. Since Recursive Advisors currently only support non-streaming calls, you can’t apply this retry pattern directly to a streamed endpoint without redesigning that flow.

Q#5. How do you prevent an infinite retry loop with a self-refining advisor?

Always set a hard maxAttempts limit and a clear fallback (typically returning the last response and logging the failure) for when the judge never approves within that limit.

Q#6. What’s the tradeoff between Direct Assessment and Pairwise Comparison judging?

Direct Assessment scores one response independently, which is cheap and works well for a live retry loop. Pairwise Comparison judges which of two responses is better, which is more reliable for comparing prompt or model variants but doesn’t map cleanly onto a single-response retry decision.

Frequently Asked Questions (FAQs)

Q#1. What is LLM-as-a-Judge in Spring AI?

It’s a pattern, implemented through Spring AI’s Recursive Advisors, where a CallAdvisor uses a separate judge model to score a generated response and automatically retries with feedback if the score falls below a threshold.

Q#2. Is LLM-as-a-Judge the same as Spring AI’s Evaluation Testing?

No. Evaluation Testing (RelevancyEvaluator, FactCheckingEvaluator) runs in JUnit before deploy. LLM-as-a-Judge runs live, inside the request path, and can retry a response automatically.

Q#3. Can I use LLM-as-a-Judge with streaming chat responses?

Not currently. Recursive Advisors in Spring AI are non-streaming only as of the 2.0 release, so this pattern targets synchronous .call() flows.

Q#4. Which Spring AI version supports Recursive Advisors?

They were introduced as an experimental feature in Spring AI 1.1.0-M4 and remain experimental through the 2.0 line.

Q#5. Does LLM-as-a-Judge increase my LLM API costs?

Yes. Every retry adds one judge call and one regeneration call, so a request that fails the judge repeatedly can cost several times a normal single-pass request.

Q#6. What judge model should I use for LLM-as-a-Judge in Spring AI?

A separate model from your generation model, ideally one benchmarked for judging tasks on a resource like the Judge Arena Leaderboard, run at low or zero temperature for consistent scoring.

Conclusion

Evaluation testing and LLM-as-a-Judge answer two different questions: did this break in CI, and is this specific live response good enough to ship right now. Spring AI’s Recursive Advisors make the second one practical, at the cost of extra latency and LLM calls you should reserve for the endpoints where a bad answer is genuinely expensive.

If you haven’t already, read the companion Spring AI Evaluation Testing guide to get the JUnit-level checks in place first. That foundation is what tells you whether a live judge advisor is even solving a problem you still have.

LLM-as-a-Judge in Spring AI: A Practical Guide