How To Build RAG With Spring AI And Pgvector

How to Build RAG with Spring AI and pgvector: Full Tutorial 2026

Every large language model has a hard cutoff. Ask it about your company’s internal onboarding policy, a document uploaded yesterday, or last week’s product release, and it either admits it does not know or, worse, makes something up that sounds confident and correct.

Your LLM cannot answer questions about last week’s internal policy update, your product’s latest release notes, or a PDF sitting in your company’s document store. It only knows what it was trained on. RAG fixes that by handing the model your own data at query time, and Spring AI makes the whole pipeline buildable in plain Java.

What You Will Learn

A Retrieval-Augmented Generation application built with Spring AI and PostgreSQL pgvector lets a Spring Boot service answer questions using your own documents. It stores text as vector embeddings inside Postgres, retrieves the closest matches for a user’s question, and feeds them to an LLM as grounded context.

By the end of this tutorial, you will have:

A Postgres instance running the pgvector extension in Docker
A Spring Boot 4 service that ingests PDFs, text, or Word files into a vector store
A working RAG query endpoint backed by Spring AI’s QuestionAnswerAdvisor
A clear picture of the mistakes that quietly break RAG pipelines in production

Prerequisites

Before you start, you need:

Java 21 or later
Maven or Gradle
Docker and Docker Compose
Basic familiarity with Spring Boot (@RestController, @Service, dependency injection)
An OpenAI API key, or a local Ollama installation if you would rather run everything offline

What Is RAG in AI?

RAG (Retrieval-Augmented Generation) is an AI technique where a language model looks up relevant information from an external source before answering, instead of relying only on what it learned during training. This keeps answers current, specific to your data, and grounded in something real.

The name RAG comes from a 2020 research paper. The idea behind the term stuck for good reason: it describes exactly what happens. Retrieve relevant information, then augment the model’s prompt with it before generation happens.

For more detail on RAG, kindly go through a separate detailed article on What is RAG in AI.

Why Does RAG Matter for Spring Boot Developers?

Without RAG, you have two expensive options for keeping an LLM current: retrain or fine-tune it every time your data changes, or cram everything into the prompt and hope it fits inside the context window. Both are slow, costly, and hard to maintain.

RAG sidesteps both problems. You keep your data in a searchable store, retrieve only the handful of relevant pieces per question, and let the model reason over just that slice. Update the underlying documents and the next query already reflects the change, with no retraining step at all.

In a production system I have worked on, the retrieval step was consistently the part that determined whether users trusted the answers, not the model itself. A great LLM given the wrong three paragraphs will still produce a confident, wrong answer.

In a production Spring Boot 4 deployment feeding a support-ticket knowledge base, retrieval itself typically stays under 50ms once the HNSW index is warm. The real bottleneck is almost always ingestion throughput, since embedding calls hit provider rate limits long before the vector search does.

What Is pgvector and How Does It Store Vectors in Postgres?

pgvector is an open-source PostgreSQL extension that adds a native vector column type and distance operators (cosine, L2, inner product) directly to Postgres. Instead of running a separate vector database, you store embeddings as rows in a normal table and query them with SQL, which means your vector data can sit next to your transactional data and even be joined in one query.

Spring AI talks to pgvector through the PgVectorStore implementation of its VectorStore interface, so your application code never touches raw SQL for similarity search. You call vectorStore.add() and vectorStore.similaritySearch(), and Spring AI handles the embedding calls and the underlying queries.

How to Build RAG with Spring AI and pgvector?

Step#1: Run PostgreSQL with pgvector in Docker

Use the official pgvector image so the extension is already compiled in.

# docker-compose.yml
services:
  postgres:
    image: pgvector/pgvector:pg16
    container_name: rag-postgres
    environment:
      POSTGRES_DB: ragdb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

Start it with:

bash
docker compose up -d

Then enable the extension once, inside the database:

sql
CREATE EXTENSION IF NOT EXISTS vector;

Step#2: Add the Spring AI Dependencies

Import the Spring AI BOM first so all starter versions stay aligned, then add the model and vector store starters.


    
        
            org.springframework.ai
            spring-ai-bom
            1.1.0
            pom
            import
        
    



    
        org.springframework.ai
        spring-ai-starter-model-openai
    
    
        org.springframework.ai
        spring-ai-starter-vector-store-pgvector
    
    
        org.springframework.boot
        spring-boot-starter-web
    
    
        org.postgresql
        postgresql
        runtime

If you want to run entirely offline, swap spring-ai-starter-model-openai for spring-ai-starter-model-ollama and point it at a local model instead.

Step#3: Configure the PgVectorStore

# application.yml
spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/ragdb
    username: postgres
    password: postgres

  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      embedding:
        options:
          model: text-embedding-3-small
      chat:
        options:
          model: gpt-4o-mini

    vectorstore:
      pgvector:
        initialize-schema: true
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        dimensions: 1536

initialize-schema: true is fine for local development since Spring AI creates the vector table for you. In a real deployment, generate that DDL once through a migration tool such as Flyway and set this back to false, so a redeploy cannot silently alter your production schema.

Step#4: Build the Document Ingestion Pipeline

Every RAG pipeline has two phases: ingesting documents once, and answering queries repeatedly. Start with ingestion.

@Service
public class DocumentIngestionService {

    private final VectorStore vectorStore;

    public DocumentIngestionService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    public void ingest(Resource resource) {
        TikaDocumentReader reader = new TikaDocumentReader(resource);
        List documents = reader.get();

        TokenTextSplitter splitter = new TokenTextSplitter();
        List chunks = splitter.apply(documents);

        vectorStore.add(chunks);
    }
}

TikaDocumentReader handles PDF, DOCX, and HTML out of the box using Apache Tika under the hood. TokenTextSplitter breaks large documents into chunks small enough to embed cleanly, with a sensible default overlap so context is not lost at chunk boundaries.

Step#5: Build the RAG Query Service

This is where retrieval and generation meet. Spring AI’s QuestionAnswerAdvisor handles the embedding of the user’s question, the similarity search, and the prompt augmentation in one call.

@Service
public class RagQueryService {

    private final ChatClient chatClient;

    public RagQueryService(ChatClient.Builder builder, VectorStore vectorStore) {
        this.chatClient = builder
                .defaultAdvisors(new QuestionAnswerAdvisor(vectorStore))
                .build();
    }

    public String ask(String question) {
        return chatClient.prompt()
                .user(question)
                .call()
                .content();
    }
}

Behind this one method call, Spring AI embeds the question, runs a similarity search against pgvector, injects the top matching chunks into the prompt, and sends the augmented prompt to the chat model.

Step#6: Expose the REST Endpoints

@RestController
@RequestMapping("/api/rag")
public class RagController {

    private final DocumentIngestionService ingestionService;
    private final RagQueryService ragQueryService;

    public RagController(DocumentIngestionService ingestionService, RagQueryService ragQueryService) {
        this.ingestionService = ingestionService;
        this.ragQueryService = ragQueryService;
    }

    @PostMapping("/ingest")
    public ResponseEntity ingest(@RequestParam("file") MultipartFile file) throws IOException {
        ingestionService.ingest(new ByteArrayResource(file.getBytes()));
        return ResponseEntity.ok("Document ingested successfully");
    }

    @GetMapping("/ask")
    public ResponseEntity ask(@RequestParam String question) {
        return ResponseEntity.ok(ragQueryService.ask(question));
    }
}

Step#7: Test the Full Pipeline

Upload a document first:

bash
curl -X POST http://localhost:8080/api/rag/ingest \
  -F "[email protected]"

Then ask a question grounded in that document:

bash
curl "http://localhost:8080/api/rag/ask?question=How many vacation days do new employees get?"

If pgvector is configured correctly, the answer comes back based on the actual handbook text, not a generic guess from the model’s training data.

RAG Architecture Patterns: Which One Are You Actually Building?

The pipeline you just built is what’s called naive RAG: one retrieval pass, one generation call, no steps in between. It works well for a single, clean document source like an employee handbook, but it’s worth knowing where it sits on the spectrum so you can recognize when to move up.

Advanced RAG adds work on either side of the retrieval step. Before retrieval, that means rewriting the user’s question into a better search query (a vague “what’s our policy” becomes “vacation day policy for new employees”). After retrieval, it means reranking the top matches with a cross-encoder before they reach the model, since vector similarity alone often ranks a loosely related chunk above a precisely relevant one.

Modular or agentic RAG goes further still, splitting retrieval across multiple specialized steps: one pass over a FAQ index, another over raw contracts, a validation step that checks whether the retrieved chunks actually answer the question before generation even runs. This is overkill for the handbook example above, but it’s the pattern to reach for once you’re serving several distinct document types through one endpoint.

A practical rule: stay with naive RAG until you notice specific failures (wrong document type retrieved, right document but wrong section), then add exactly the layer that fixes that failure. Bolting on reranking or query rewriting before you’ve seen a real failure usually just adds latency for no measurable gain.

Choosing a Chunking Strategy That Actually Improves Retrieval

TokenTextSplitter in this tutorial uses fixed-size chunking: it counts tokens and cuts at that boundary, regardless of what sentence or section it lands in. That’s fine for a first pass, but it’s the single easiest thing to improve once retrieval quality feels inconsistent.

Fixed-size chunking is fast and predictable but can split a sentence, or worse, a table row, right down the middle. Recursive chunking tries paragraph breaks first, then sentence breaks, only falling back to a hard token cut if a paragraph is still too long. It costs almost nothing extra to compute and produces noticeably cleaner chunk boundaries. Semantic chunking goes further, using embedding similarity between adjacent sentences to decide where a topic actually shifts, which is worth the extra cost for long, dense documents like contracts but is unnecessary overhead for short FAQ-style content.

Document structure matters more than the splitter you pick. A handbook with clear headings retrieves far better if you chunk along those headings first and only then apply token limits within each section, rather than splitting blindly from the top of the file. If you’re ingesting mixed content types, such as short FAQ entries alongside long policy documents, consider running two different chunking configurations rather than forcing one splitter setting to serve both well.

How to Measure Whether Your RAG Pipeline Is Actually Working

It’s easy to ship a RAG endpoint that returns fluent, well-formatted answers that are quietly wrong, because a language model will write a confident sentence out of almost any context you hand it. Testing this by eyeballing a few sample questions isn’t enough once real users start asking things you didn’t anticipate.

Two categories of metrics matter here, and they catch different failures. Retrieval metrics (precision@k and recall@k) tell you whether the chunks coming back from vectorStore.similaritySearch() actually contain the answer at all, independent of what the LLM later does with them. Generation metrics, often grouped under “faithfulness” or “groundedness,” check whether the model’s final answer sticks strictly to the retrieved context or drifts into something it just knows from training. A pipeline can fail at either stage, and they need separate diagnosis: bad retrieval means fixing chunking or embeddings, bad faithfulness means tightening the prompt template.

Open-source frameworks like RAGAS automate both categories against a labeled test set, but you don’t need tooling to start. A simple manual habit works well early on: log the retrieved chunks alongside every answer during testing, and once a week, spot-check ten random pairs and ask two questions of each: did the right chunk come back, and did the model actually use it? That single habit catches most retrieval problems well before real users notice them.

Common Mistakes to Avoid

Forgetting CREATE EXTENSION vector: Spring AI cannot create the vector column type on its own; the extension must exist in the database first, or every insert fails with a cryptic type error.
Dimension mismatch: If your embedding model outputs 1536 dimensions but your dimensions property says something else, inserts fail silently or return garbage similarity scores. Always match the two exactly.
Skipping the HNSW index: Without it, pgvector falls back to a sequential scan, and similarity search gets noticeably slower as your document count grows past a few thousand chunks.
Chunking too large: Chunks of 1,500+ tokens dilute the relevance signal; the retriever pulls back a chunk that is only partially about the question, and the model has to guess at the rest.
Leaving initialize-schema: true in production: It is convenient in development but risky once real data and real migrations are involved.

How Do Interviewers Test RAG Knowledge?

Q: What problem does RAG solve that fine-tuning does not?

A: RAG keeps the model’s answers current without retraining. You update the vector store, not the model weights, so new documents are searchable within seconds of ingestion.

Q: Why use cosine distance instead of Euclidean distance for text embeddings?

A: Text embeddings are typically normalised, so cosine distance measures the angle between vectors rather than raw magnitude, which better reflects semantic similarity for language models.

Q: What is the tradeoff of an HNSW index versus IVFFlat in pgvector?

A: HNSW gives faster, more accurate approximate search at a higher memory and build-time cost, while IVFFlat builds faster and uses less memory but needs a training step and is more sensitive to the number of lists you configure.

Q: What does the QuestionAnswerAdvisor actually do under the hood?

A: It intercepts the outgoing chat prompt, runs a similarity search against the configured VectorStore, and rewrites the prompt to include the retrieved chunks as context before the call reaches the model.

Frequently Asked Questions (FAQs)

What is RAG in Spring AI?

RAG in Spring AI is a pattern where the VectorStore abstraction retrieves relevant document chunks for a user’s question, and an advisor like QuestionAnswerAdvisor injects them into the prompt sent to the chat model.

Does pgvector need a separate vector database?

No. pgvector is a Postgres extension, so if you already run Postgres, you do not need to introduce a dedicated vector database like Pinecone or Milvus for most workloads.

Which embedding model works best with Spring AI RAG?

It depends on your budget and privacy needs. OpenAI’s text-embedding-3-small is a strong, low-cost default, while nomic-embed-text through Ollama works well if you need everything to run fully offline.

How do I choose chunk size for RAG in Spring Boot?

Start around 300 to 800 tokens per chunk with some overlap, then adjust based on how coherent your retrieved answers are. Smaller chunks improve precision; larger chunks preserve more surrounding context.

Can I use RAG with local LLMs instead of OpenAI?

Yes. Swap the OpenAI starter for the Ollama starter and point Spring AI at a local model such as Llama 3.2; the VectorStore and advisor code stays exactly the same.

Is pgvector fast enough for production RAG?

Yes, for most application-scale workloads. With an HNSW index, pgvector handles millions of vectors with sub-100ms similarity search, and Postgres’s existing operational tooling (backups, replication, monitoring) comes along for free.

Conclusion

You now have a working Spring Boot RAG pipeline: ingestion through TikaDocumentReader and TokenTextSplitter, storage in PgVectorStore, and retrieval through QuestionAnswerAdvisor. The pattern scales from a single PDF to a full internal knowledge base without changing the core code.

Next, look at adding conversational memory so the chat client remembers earlier turns in the same session, and consider a re-ranking step if your retrieved chunks start feeling only loosely relevant. If you have not already, check out our companion article on building an MCP Server with Spring Boot to see how the same Spring AI foundation extends to tool-calling for external AI clients.

References:

Spring AI RAG

Spring AI VectorStore interface

pgvector extension documentation

How to Build RAG with Spring AI and pgvector