Introduction

Every post in this series has quietly touched a piece of the same problem. Building Agentic Workflows in Java said toolUse.input() is untrusted and must be validated before it reaches your code. Building Reliable LLM Applications in Java said the model will confidently invent facts, so ground it and get typed output instead of parsing prose. Neither post named the thing underneath both statements: anything that crosses from outside your code into the model, or from the model back into your code, is untrusted input — a request body from the network, not a trusted internal value. This post names that boundary directly and gathers the defenses in one place — prompt injection (direct and indirect), input validation, output validation, and PII redaction — with the SAFE pattern shown beside every unsafe one it replaces, since this is the security-forward capstone of the series.

Introduction

Building Reliable LLM Applications in Java put it plainly: treat model output as a hypothesis to verify, not a fact to trust. Testing Best Practices in Java put the same discipline in JUnit terms: a suite only earns trust by asserting the right things at the right level, unhappy paths included. This post is where those two ideas meet — a JUnit test either passes or fails against a fixed expected value; an LLM’s output is a paragraph of prose that might be right in spirit while differing token-for-token from anything you wrote down in advance. Evaluating it takes a harness, not an assertEquals.

Introduction

Every LLM ecosystem now has at least one framework promising to make agents easier to build, and every framework post either oversells the abstraction or dismisses it outright. Neither is useful. The only honest way to evaluate a framework is to build the same thing twice — once on the vendor’s raw SDK, once on the framework — and compare what each version actually cost you and actually gave you back.

Introduction

RAG From Scratch in Java built a retrieval pipeline out of cosine similarity and a reranking pass, and Vector Databases in Practice for Java moved that same index into pgvector so it can hold millions of chunks. Neither post asked the question that actually decides whether a RAG system is any good: does it retrieve the right chunks, and how would you know?

This post answers that question in two halves. First, three techniques that improve what gets retrieved in the first place — hybrid search that catches what pure vector similarity misses, metadata filtering that narrows the search space before ranking even starts, and chunking choices that shape recall long before a query is ever run. Second, the metrics that turn “this feels better” into a number you can track across a change: recall@k, precision@k, MRR, and nDCG. Everything below is illustrative, non-executed prose code, consistent with the pipeline built in post 20.

Introduction

We already covered picking the right model tier for the task and caching a large shared prefix in https://pg-blogs.netlify.app/posts/11-building-reliable-llm-apps-in-java/. Those two lines were the tip of a bigger discipline: LLM cost is not a fixed line item, it’s an engineering variable — one you can measure and shrink with the same rigor you’d apply to database query time or container memory.

This post goes deeper: how input/output pricing actually works, the exact cache_control shape and how to prove a cache hit rather than assume one, the Batches API for work that isn’t latency-sensitive, and model routing — using a cheap model to triage, escalating only the hard cases to a stronger one. The honest framing throughout: measure before you optimize. Every technique here has a cost of its own; applied to the wrong workload, “optimization” makes things slower or more expensive.

Introduction

Retrieval-augmented generation (RAG) is the pattern behind almost every “chat with your documents” feature: instead of hoping a model already knows the answer, you find the passages most likely to contain it and hand them to the model as context. Done well, it’s the single highest-leverage technique for keeping an LLM application grounded in facts it didn’t memorize.

This post builds RAG from scratch — no vector database, no framework — so the mechanics are visible end to end: splitting documents into chunks, turning them into embeddings, searching them with nothing more than array math, reranking the results, and generating a grounded answer. It builds directly on the grounding discipline from Building Reliable LLM Applications in Java — “give the model the source material and instruct it to answer only from that material” — by showing where that source material actually comes from. We’ll close with the question every RAG design eventually has to answer honestly: when does retrieval beat simply pasting more into the context window?

Introduction

Every agent needs tools, and every tool needs a way to reach the model. Building Agentic Workflows in Java built that connection by hand — a hand-written Tool schema, a loop that dispatches on toolUse.name(). LLM Frameworks vs. the Raw SDK in Java showed LangChain4j and Spring AI turning an annotated Java method into that same schema via reflection. Both are still bespoke: the tool lives inside one process, wired to one agent, in one language.

Introduction

RAG From Scratch in Java built retrieval with nothing but an array of doubles and a Comparator: cosine similarity computed in a loop, top-k picked with a stream sort. That post said outright that this is a brute-force O(n) scan — fine for a few thousand chunks, the wrong tool once a corpus reaches millions. This post picks up exactly there: how do you store and search vectors at that scale, using Postgres, and when do you need something else entirely?

Introduction

An ORM like JPA/Hibernate makes database access feel like working with plain objects. That is exactly the problem: a single innocent-looking loop can fire hundreds of SQL queries, and you won’t see it until production slows to a crawl.

This post covers the ORM traps that bite Java teams most often — starting with the notorious N+1 query problem — and the concrete techniques that fix them.

The N+1 Problem

Consider loading authors and printing their books:

Introduction

“Agent” has become the word for any program that calls an LLM more than once, which makes it a word worth being precise about. An agent, in the sense this post uses, is a loop: the model decides which tool to call next, your code executes it, and the result feeds back in — repeating until the model decides it’s done. That’s a genuinely different (and riskier) shape than a single request/response call.