Introduction

Retrieval-augmented generation (RAG) is the pattern behind almost every “chat with your documents” feature: instead of hoping a model already knows the answer, you find the passages most likely to contain it and hand them to the model as context. Done well, it’s the single highest-leverage technique for keeping an LLM application grounded in facts it didn’t memorize.

This post builds RAG from scratch — no vector database, no framework — so the mechanics are visible end to end: splitting documents into chunks, turning them into embeddings, searching them with nothing more than array math, reranking the results, and generating a grounded answer. It builds directly on the grounding discipline from Building Reliable LLM Applications in Java — “give the model the source material and instruct it to answer only from that material” — by showing where that source material actually comes from. We’ll close with the question every RAG design eventually has to answer honestly: when does retrieval beat simply pasting more into the context window?

Introduction

Retrieval-augmented generation (RAG) is the pattern behind almost every “chat with your documents” feature: instead of hoping a model already knows the answer, you find the passages most likely to contain it and hand them to the model as context. Done well, it’s the single highest-leverage technique for keeping an LLM application grounded in facts it didn’t memorize.

This post builds RAG from scratch — no vector database, no framework — so the mechanics are visible end to end: splitting documents into chunks, turning them into embeddings, searching them with nothing more than array math, reranking the results, and generating a grounded answer. It builds directly on the grounding discipline from Building Reliable LLM Applications in Python — “give the model the source material and instruct it to answer only from that material” — by showing where that source material actually comes from. We’ll close with the question every RAG design eventually has to answer honestly: when does retrieval beat simply pasting more into the context window?

Introduction

Every agent needs tools, and every tool needs a way to reach the model. Building Agentic Workflows in Java built that connection by hand — a hand-written Tool schema, a loop that dispatches on toolUse.name(). LLM Frameworks vs. the Raw SDK in Java showed LangChain4j and Spring AI turning an annotated Java method into that same schema via reflection. Both are still bespoke: the tool lives inside one process, wired to one agent, in one language.

Introduction

Every agent needs tools, and every tool needs a way to reach the model. Building Agentic Workflows in Python built that connection by hand — a hand-written JSON schema, a loop that dispatches on block.name. LLM Frameworks vs. the Raw SDK in Python showed LangChain’s @tool turning a plain function into that same schema via bind_tools. Both are still bespoke: the tool lives inside one process, wired to one agent, in one language.

Introduction

RAG From Scratch in Java built retrieval with nothing but an array of doubles and a Comparator: cosine similarity computed in a loop, top-k picked with a stream sort. That post said outright that this is a brute-force O(n) scan — fine for a few thousand chunks, the wrong tool once a corpus reaches millions. This post picks up exactly there: how do you store and search vectors at that scale, using Postgres, and when do you need something else entirely?

Introduction

RAG From Scratch in Python built retrieval with nothing but a list of floats and sorted(): cosine similarity computed in a loop, top-k picked with a slice. That post said outright that this is a brute-force O(n) scan — fine for a few thousand chunks, the wrong tool once a corpus reaches millions. This post picks up exactly there: how do you store and search vectors at that scale, using Postgres, and when do you need something else entirely?

Introduction

“Agent” has become the word for any program that calls an LLM more than once, which makes it a word worth being precise about. An agent, in the sense this post uses, is a loop: the model decides which tool to call next, your code executes it, and the result feeds back in — repeating until the model decides it’s done. That’s a genuinely different (and riskier) shape than a single request/response call.

Introduction

“Agent” has become the word for any program that calls an LLM more than once, which makes it a word worth being precise about. An agent, in the sense this post uses, is a loop: the model decides which tool to call next, your code executes it, and the result feeds back in — repeating until the model decides it’s done. That’s a genuinely different (and riskier) shape than a single request/response call.

Introduction

LLMs are usually associated with Python, but a great deal of production software — banking, enterprise backends, long-lived services — runs on the JVM, and those systems increasingly need to call language models too. Java’s strong typing and mature tooling are genuine assets here: they push you toward exactly the discipline reliable LLM applications require.

The core mindset is the same in any language: treat model output as a hypothesis to verify, not a fact to trust. This post covers the practices that make Java LLM applications production-grade, using Anthropic’s Claude and the official anthropic-java SDK.

Introduction

Calling an LLM API is easy. Building an application on top of one that is reliable — that fails predictably, doesn’t hallucinate its way into wrong answers, and doesn’t surprise you with a bill — is a real engineering discipline.

The core mindset shift: treat model output as a hypothesis to verify, not a fact to trust. This post covers the practices that make Python LLM applications production-grade, using Anthropic’s Claude and the official anthropic SDK.