Spring AI Evaluation Testing: How to Test LLM Responses

Large Language Models (LLMs) like OpenAI GPT, Gemini, Claude, and Llama have transformed how modern applications generate content, answer questions, summarize documents, and even write code. With the introduction of Spring AI, integrating these powerful AI models into Spring Boot applications has become much easier for Java developers.

However, there’s one important question every developer should ask:

Unlike traditional software, LLMs don’t always produce the same output for the same input. Their responses can vary in wording, accuracy, completeness, and even contain incorrect or misleading information (often called hallucinations). Because of this, simply checking whether your API returns a response is no longer enough.

LLMs never give you the same sentence twice, so assertEquals is useless the moment your Spring Boot app starts calling a chat model. Spring AI ships a proper answer to this: an Evaluator API with two ready-made implementations, RelevancyEvaluator and FactCheckingEvaluator, built specifically for asserting on AI output in JUnit. This guide walks through both, plus how to run the tests locally without an API bill.

In this tutorial, we’ll explore Spring AI Evaluation Testing from the ground up. Even if you’re completely new to AI testing, you’ll learn how to evaluate LLM responses using practical Spring Boot examples, understand the available evaluators, and write automated tests that ensure your AI-powered applications are both accurate and reliable.

What Is Spring AI Evaluation Testing?

Spring AI Evaluation Testing is the practice of using the framework’s Evaluator interface, along with its RelevancyEvaluator and FactCheckingEvaluator implementations, to have one AI model judge whether another model’s response stayed on-topic and stuck to the facts, so JUnit tests can assert on quality instead of exact text.

That single idea, letting a model grade a model, changes how you write tests for anything backed by a ChatClient. Instead of writing a test that breaks the moment the wording changes, you write one that checks the property you actually care about: did the answer address the question, and did it stay true to the source material.

What You Will Learn

• What the Evaluator, EvaluationRequest, and EvaluationResponse types actually do
• How to wire RelevancyEvaluator into a JUnit test for a RAG-based Spring Boot app
• How to use FactCheckingEvaluator to catch hallucinated claims before they ship
• How to run both evaluators locally, against a containerized model, with zero API cost
• Where LLM-as-a-Judge and Recursive Advisors fit once testing turns into a live production quality gate

Prerequisites

• Working knowledge of Spring Boot and JUnit 5
• A Spring Boot project on Spring Boot 4.1 with Spring AI 2.0 (Spring AI 2.0 also runs on Spring Boot 4.0; JUnit 4 support was removed in Boot 4, so make sure any legacy test classes are already on Jupiter)
• Docker installed and running, for Testcontainers
• Basic familiarity with Retrieval Augmented Generation (RAG) concepts such as a VectorStore and QuestionAnswerAdvisor

Why is Spring AI Evaluation Testing mandatory to know?

Imagine you’re building an AI-powered customer support chatbot. If a customer asks:

“What is the purpose of a Circuit Breaker in Microservices?”

You expect the AI to provide an accurate and meaningful explanation. But what if the response is incomplete, factually incorrect, or completely unrelated? Your application may still work technically, but it won’t deliver a good user experience.

This is where AI Evaluation Testing becomes essential.

AI evaluation testing verifies the quality of an LLM’s responses rather than just checking whether the API call succeeded. It helps you automatically measure whether the generated answer is:

• Correct and factually accurate
• Relevant to the user’s question
• Complete and informative
• Free from hallucinations
• Safe and aligned with your application’s requirements

Instead of manually reading every AI response, you can automate these quality checks as part of your testing process.

Fortunately, Spring AI provides a built-in Evaluation Framework that makes this straightforward. It allows you to write automated tests that compare AI-generated responses against expected results using different evaluation strategies. This gives you confidence that your AI application continues to produce high-quality answers as prompts, models, or datasets evolve.

Why Do LLM Responses Need Automated Evaluation?

A chat model can answer the same prompt differently on every run, even at low temperature. That breaks the entire premise of a unit test that expects a fixed string. It also hides a second, more dangerous problem: a response can read fluently and still be wrong, off-topic, or flatly contradict the document it was supposed to be grounded in.

In a production RAG deployment, teams typically discover this the hard way: a support bot answers confidently and politely, and the answer is still wrong, because nothing in the pipeline ever checked the response against the source document. Evaluation testing exists to catch exactly that, automatically, before it reaches a user.

Spring AI’s answer is to treat evaluation as its own testable concern, separate from generation, using another model as the judge.

How to do Spring AI Evaluation Testing?

Step#1: Understand the Evaluator Interface, EvaluationRequest, and EvaluationResponse

Everything in Spring AI’s evaluation testing support flows through one functional interface:

package org.springframework.ai.evaluation;

@FunctionalInterface
public interface Evaluator {
    EvaluationResponse evaluate(EvaluationRequest evaluationRequest);
}

You feed it an EvaluationRequest, built from three pieces of data:

• userText: the original question or claim you’re checking
• dataList: the supporting context, typically the documents your RAG pipeline retrieved
• responseContent: the AI-generated text you want to grade

It hands back an EvaluationResponse, and the only method you’ll call in most tests is isPass(), a boolean that tells you whether the judge model considered the response acceptable.

Spring AI ships two concrete evaluators against that interface: RelevancyEvaluator and FactCheckingEvaluator. They look similar on paper but check for different failure modes, which is exactly where most teams get their test assertions wrong. More on that in the Common Mistakes section below.

Full reference: Spring AI Model Evaluation documentation.

Step#2: Add Spring AI Evaluation Dependencies to Your Project

The evaluators live in Spring AI’s core, so no separate starter is required for them. You do need a chat model starter to actually run the evaluation, plus the Testcontainers integration if you want to run everything locally (covered in Step 5).

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>2.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-ollama</artifactId>
    </dependency>

    <!-- Testing -->
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-spring-boot-testcontainers</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>ollama</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>junit-jupiter</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

This example uses Ollama for both the app and the tests, which keeps the whole tutorial runnable without an API key. If your production app calls a hosted model like OpenAI or Anthropic, swap that starter in and keep Ollama for the test scope only, since a small local model is normally all a judge needs.

Step#3: How Does the RelevancyEvaluator Work?

RelevancyEvaluator checks one narrow question: given the retrieved context, does the response actually address what the user asked? It doesn’t check whether the answer is correct, only whether it’s on-topic.

Picture a support bot answering questions from a knowledge base:

@SpringBootTest
@Testcontainers
class SupportBotRelevancyTest {

    @Container
    @ServiceConnection
    static OllamaContainer ollama =
            new OllamaContainer("ollama/ollama:0.6.8").withReuse(true);

    @Autowired
    private ChatModel chatModel;

    @Autowired
    private VectorStore vectorStore;

    @Test
    void answerShouldStayOnTopic() {
        String question = "What is our refund policy for annual plans?";

        ChatClient chatClient = ChatClient.builder(chatModel).build();

        ChatResponse chatResponse = chatClient.prompt(question)
                .advisors(new QuestionAnswerAdvisor(vectorStore))
                .call()
                .chatResponse();

        String answer = chatResponse.getResult().getOutput().getText();

        EvaluationRequest request = new EvaluationRequest(
                question,
                (List<Content>) chatResponse.getMetadata()
                        .get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS),
                answer
        );

        RelevancyEvaluator relevancyEvaluator =
                new RelevancyEvaluator(ChatClient.builder(chatModel));

        EvaluationResponse result = relevancyEvaluator.evaluate(request);

        assertThat(result.isPass())
                .as("Answer should address: %s", question)
                .isTrue();
    }
}

Note the argument order in EvaluationRequest: the question goes first, the retrieved documents second, the generated answer third. RelevancyEvaluator reads it in exactly that order: query, context, response.

Step#4: How Does FactCheckingEvaluator Catch Hallucinations?

FactCheckingEvaluator asks a different question: is this specific claim actually supported by this specific document? It’s the tool for catching hallucinations, where the model states something confidently that the source material never said.

@Test
void answerShouldNotContradictPolicyDocument() {
    String policyDocument = """
            Annual plan subscribers can request a full refund within 30 days
            of purchase. Monthly plan subscribers are not eligible for refunds
            after the billing cycle starts.
            """;

    String hallucinatedClaim =
            "Annual subscribers can get a refund any time within 90 days.";

    FactCheckingEvaluator factCheckingEvaluator =
            new FactCheckingEvaluator(ChatClient.builder(chatModel));

    EvaluationRequest request = new EvaluationRequest(
            policyDocument,
            Collections.emptyList(),
            hallucinatedClaim
    );

    EvaluationResponse result = factCheckingEvaluator.evaluate(request);

    assertThat(result.isPass())
            .as("A wrong refund window should fail fact-checking")
            .isFalse();
}

Look closely at that EvaluationRequest and compare it with Step 3’s. For FactCheckingEvaluator, the document goes in the userText slot and the claim goes in responseContent, with an empty context list in between. It’s the same constructor, but the two evaluators assign completely different meaning to those three positions. Copy-pasting one evaluator’s request shape into the other is the single most common bug teams hit with this API.

For the judge model itself, Spring AI’s docs point to Bespoke’s bespoke-minicheck, a small model purpose-built for fact-checking that’s noticeably cheaper to run at scale than a general-purpose model like GPT-4, and it’s available through Ollama.

Step#5: Run Evaluator Tests Locally with Testcontainers and Ollama

Running evaluation tests against a hosted API on every CI build gets slow and expensive fast, since every test now makes at least two model calls: one to generate, one to judge. Spring AI’s spring-ai-spring-boot-testcontainers module solves this by auto-configuring the connection to a container for you.

The @ServiceConnection annotation used in Step 3 is doing real work: it matches the OllamaContainer type against Spring AI’s built-in OllamaConnectionDetails, so spring.ai.ollama.base-url gets wired automatically. No @DynamicPropertySource boilerplate needed. Full details: Spring AI Testcontainers reference.

One practical wrinkle: the container starts empty, so the model has to be pulled before the first test runs.

@BeforeAll
static void pullModel() throws IOException, InterruptedException {
    ollama.execInContainer("ollama", "pull", "llama3.2");
}

Pull it once in a @BeforeAll, or bake a custom image with the model already included, so your CI pipeline isn’t re-downloading it on every run.

Step#6: Go Further with LLM-as-a-Judge and Recursive Advisors

Once evaluation tests are working, the same idea can move from test code into the running application. Spring AI’s LLM-as-a-Judge guide documents this as a Recursive Advisor: a CallAdvisor that scores a response after generation and automatically retries with the judge’s feedback if the score falls short, all before the response reaches the caller.

The pattern deliberately reuses a separate, cheaper model as the judge, the same “don’t let a model grade its own homework” principle behind FactCheckingEvaluator’s recommendation to use Bespoke-Minicheck instead of the generation model. This is genuinely experimental in current Spring AI releases and worth knowing about, but treat your JUnit-level RelevancyEvaluator and FactCheckingEvaluator tests as the foundation before reaching for a live retry loop in production.

Common Mistakes to Avoid

• Swapping the EvaluationRequest argument order. RelevancyEvaluator expects (question, context, answer). FactCheckingEvaluator expects (document, empty context, claim). Mixing these up silently produces meaningless scores instead of a compile error.
• Judging a model with itself. Using the same ChatModel instance and prompt style for generation and evaluation risks the model rating its own output more favorably than it deserves.
• Skipping temperature control on the judge. Leave the evaluator model’s temperature at its default and the same test can pass on one CI run and fail on the next for no code-related reason. Pin it to zero.
• Assuming JUnit 4 annotations still work. Spring Boot 4 dropped JUnit 4 entirely. Any older evaluator test copied from a pre-2026 blog post needs Jupiter’s @Test, not org.junit.Test.
• Hitting a paid API on every build. Running both generation and evaluation against a hosted model on every commit adds up fast. Reserve that for a smaller nightly suite and run the bulk of your evaluator tests against a local model via Testcontainers.
• Treating isPass() as infallible. The judge model can be wrong too. For high-stakes domains like billing, medical, or legal content, use evaluator tests to catch obvious regressions, not as a substitute for human review.

Interview Questions on This Topic

Q#1. What problem does Spring AI’s Evaluator interface solve?

LLM output is non-deterministic, so exact-match assertions are unreliable. Evaluator standardizes using one AI model to judge another’s response for relevancy or factual accuracy, so tests assert isPass() instead of comparing strings.

Q#2. What’s the functional difference between RelevancyEvaluator and FactCheckingEvaluator?

RelevancyEvaluator checks whether a response is on-topic for the question and context, even if it’s factually wrong. FactCheckingEvaluator checks whether a specific claim is supported by a specific document, regardless of whether it answers any question at all.

Q#3. Why use a different model to evaluate than to generate?

Reusing the same model risks narcissistic bias, where a model scores its own output more generously. Spring AI’s docs recommend a separate, often smaller, dedicated model such as Bespoke-Minicheck for fact-checking.

Q#4. How do you stop evaluation-based tests from becoming flaky?

Set the judge model’s temperature to zero, run it against a version-pinned local model through Testcontainers rather than a live API, and treat borderline scores as a reason to review the prompt rather than an immediate build failure.

Q#5. How would you avoid API costs when running evaluator tests in CI?

Use spring-ai-spring-boot-testcontainers with an OllamaContainer and @ServiceConnection, so both the generation call and the evaluation call run against a small local model instead of a metered endpoint.

Q#6. How does LLM-as-a-Judge relate to RelevancyEvaluator and FactCheckingEvaluator?

LLM-as-a-Judge is the general technique of scoring one model’s output with another model. Spring AI’s two evaluators are concrete implementations of that idea for test code, while Recursive Advisors apply the same pattern live in production to retry a weak response automatically.

Frequently Asked Questions (FAQs)

Q#1. What is Spring AI Evaluation Testing?

It’s Spring AI’s built-in support for using an AI model to judge another model’s response, through the Evaluator interface and its RelevancyEvaluator and FactCheckingEvaluator implementations, so JUnit tests can assert on response quality instead of exact wording.

Q#2. How do I test if an LLM response is factually accurate in Spring Boot?

Use FactCheckingEvaluator, passing your source document as the context and the model’s claim as the response content, then assert on EvaluationResponse.isPass() in a standard JUnit 5 test.

Q#3. Can I use Spring AI Evaluators without OpenAI?

Yes. Both evaluators only need a ChatClient.Builder, so you can point them at Ollama, Anthropic, or any chat model Spring AI supports, including a fully local Ollama model run through Testcontainers.

Q#4. Is RelevancyEvaluator only useful for RAG applications?

It’s most valuable in RAG flows since it checks the response against retrieved context, but you can pass an empty context list and still use it to check general on-topic relevance between a question and an answer.

Q#5. What Spring Boot version do I need for Spring AI 2.0?

Spring AI 2.0 targets Spring Boot 4.0 or 4.1 with Spring Framework 7. If you’re still on Spring Boot 3.x, use the Spring AI 1.x line instead.

Q#6. Do I need a GPU to run evaluation tests locally?

No. Small evaluator-focused models like Bespoke-Minicheck or a compact Llama model run acceptably on CPU through Ollama for test purposes; a GPU only becomes worthwhile if your test suite is large enough that evaluation latency starts slowing down CI.

Conclusion

Testing an LLM-backed feature is really testing two things: did it answer the question, and did it stick to the facts. Spring AI’s RelevancyEvaluator and FactCheckingEvaluator map directly onto those two checks, and running them through Testcontainers means you can wire this into CI without a per-build API bill.

Next, it’s worth pairing this with Spring AI’s Retrieval Augmented Generation support if you haven’t already built the RAG pipeline these evaluators are testing, and then circling back to Recursive Advisors once you want the same quality check running live, not just in tests.

If you found this useful, share it in your circle, and drop your own evaluator gotchas in the comments so the next reader doesn’t have to hit them the hard way.

---