Introduction

Every post in this series has quietly touched a piece of the same problem. Building Agentic Workflows in Python said a tool’s input is untrusted and must be validated before it reaches your code. Building Reliable LLM Applications in Python said the model will confidently invent facts, so ground it and get typed output instead of parsing prose. Neither post named the thing underneath both statements: anything that crosses from outside your code into the model, or from the model back into your code, is untrusted input — a request body from the network, not a trusted internal value. This post names that boundary directly and gathers the defenses in one place — prompt injection (direct and indirect), input validation, output validation, and PII redaction — with the SAFE pattern shown beside every unsafe one it replaces, since this is the security-forward capstone of the series.

Introduction

Building Reliable LLM Applications in Python put it plainly: treat model output as a hypothesis to verify, not a fact to trust. Testing Best Practices in Python put the same discipline in pytest terms: a suite only earns trust by asserting the right things at the right level, unhappy paths included. This post is where those two ideas meet — a pytest assertion either passes or fails against a fixed expected value; an LLM’s output is a paragraph of prose that might be right in spirit while differing token-for-token from anything you wrote down in advance. Evaluating it takes a harness, not an assert.

Introduction

Every LLM ecosystem now has at least one framework promising to make agents easier to build, and every framework post either oversells the abstraction or dismisses it outright. Neither is useful. The only honest way to evaluate a framework is to build the same thing twice — once on the vendor’s raw SDK, once on the framework — and compare what each version actually cost you and actually gave you back.

Introduction

RAG From Scratch in Python built a retrieval pipeline out of cosine similarity and a reranking pass, and Vector Databases in Practice for Python moved that same index into pgvector so it can hold millions of chunks. Neither post asked the question that actually decides whether a RAG system is any good: does it retrieve the right chunks, and how would you know?

This post answers that question in two halves. First, three techniques that improve what gets retrieved in the first place — hybrid search that catches what pure vector similarity misses, metadata filtering that narrows the search space before ranking even starts, and chunking choices that shape recall long before a query is ever run. Second, the metrics that turn “this feels better” into a number you can track across a change: recall@k, precision@k, MRR, and nDCG. Everything below is illustrative, non-executed prose code, consistent with the pipeline built in post 21.

Introduction

https://pg-blogs.netlify.app/posts/10-building-reliable-llm-apps-in-python/ closed with a section on picking the right model per task and caching a shared prefix. That was the entry point into a bigger discipline: LLM spend is an engineering variable, not a fixed bill — one you can measure and reduce with the same rigor you’d apply to query latency or memory footprint.

This post goes deeper on four levers: how input/output pricing actually works and why the prefix is usually where the money goes, the exact cache_control shape and how to prove a cache hit instead of assuming one, the Batches API for work that isn’t latency-sensitive, and model routing — a cheap model triaging requests and escalating only the hard ones. The throughline is honest: measure before you optimize. Every lever here has its own cost; misapplied, it makes things slower or pricier, not cheaper.

Introduction

Retrieval-augmented generation (RAG) is the pattern behind almost every “chat with your documents” feature: instead of hoping a model already knows the answer, you find the passages most likely to contain it and hand them to the model as context. Done well, it’s the single highest-leverage technique for keeping an LLM application grounded in facts it didn’t memorize.

This post builds RAG from scratch — no vector database, no framework — so the mechanics are visible end to end: splitting documents into chunks, turning them into embeddings, searching them with nothing more than array math, reranking the results, and generating a grounded answer. It builds directly on the grounding discipline from Building Reliable LLM Applications in Python — “give the model the source material and instruct it to answer only from that material” — by showing where that source material actually comes from. We’ll close with the question every RAG design eventually has to answer honestly: when does retrieval beat simply pasting more into the context window?

Introduction

Every agent needs tools, and every tool needs a way to reach the model. Building Agentic Workflows in Python built that connection by hand — a hand-written JSON schema, a loop that dispatches on block.name. LLM Frameworks vs. the Raw SDK in Python showed LangChain’s @tool turning a plain function into that same schema via bind_tools. Both are still bespoke: the tool lives inside one process, wired to one agent, in one language.

Introduction

RAG From Scratch in Python built retrieval with nothing but a list of floats and sorted(): cosine similarity computed in a loop, top-k picked with a slice. That post said outright that this is a brute-force O(n) scan — fine for a few thousand chunks, the wrong tool once a corpus reaches millions. This post picks up exactly there: how do you store and search vectors at that scale, using Postgres, and when do you need something else entirely?

Introduction

Python’s ORMs — SQLAlchemy and the Django ORM — are a joy to write and dangerously easy to make slow. A loop over a queryset that reads one related field can silently fire hundreds of queries. It looks fine in development and falls over on real data.

This post covers the ORM traps Python teams hit most, starting with the N+1 query problem, in both SQLAlchemy and Django.

The N+1 Problem

You load authors and print each author’s book count:

Introduction

“Agent” has become the word for any program that calls an LLM more than once, which makes it a word worth being precise about. An agent, in the sense this post uses, is a loop: the model decides which tool to call next, your code executes it, and the result feeds back in — repeating until the model decides it’s done. That’s a genuinely different (and riskier) shape than a single request/response call.