Introduction

We already covered picking the right model tier for the task and caching a large shared prefix in https://pg-blogs.netlify.app/posts/11-building-reliable-llm-apps-in-java/. Those two lines were the tip of a bigger discipline: LLM cost is not a fixed line item, it’s an engineering variable — one you can measure and shrink with the same rigor you’d apply to database query time or container memory.

This post goes deeper: how input/output pricing actually works, the exact cache_control shape and how to prove a cache hit rather than assume one, the Batches API for work that isn’t latency-sensitive, and model routing — using a cheap model to triage, escalating only the hard cases to a stronger one. The honest framing throughout: measure before you optimize. Every technique here has a cost of its own; applied to the wrong workload, “optimization” makes things slower or more expensive.

Introduction

https://pg-blogs.netlify.app/posts/10-building-reliable-llm-apps-in-python/ closed with a section on picking the right model per task and caching a shared prefix. That was the entry point into a bigger discipline: LLM spend is an engineering variable, not a fixed bill — one you can measure and reduce with the same rigor you’d apply to query latency or memory footprint.

This post goes deeper on four levers: how input/output pricing actually works and why the prefix is usually where the money goes, the exact cache_control shape and how to prove a cache hit instead of assuming one, the Batches API for work that isn’t latency-sensitive, and model routing — a cheap model triaging requests and escalating only the hard ones. The throughline is honest: measure before you optimize. Every lever here has its own cost; misapplied, it makes things slower or pricier, not cheaper.

Introduction

RAG From Scratch in Java built retrieval with nothing but an array of doubles and a Comparator: cosine similarity computed in a loop, top-k picked with a stream sort. That post said outright that this is a brute-force O(n) scan — fine for a few thousand chunks, the wrong tool once a corpus reaches millions. This post picks up exactly there: how do you store and search vectors at that scale, using Postgres, and when do you need something else entirely?

Introduction

RAG From Scratch in Python built retrieval with nothing but a list of floats and sorted(): cosine similarity computed in a loop, top-k picked with a slice. That post said outright that this is a brute-force O(n) scan — fine for a few thousand chunks, the wrong tool once a corpus reaches millions. This post picks up exactly there: how do you store and search vectors at that scale, using Postgres, and when do you need something else entirely?

Introduction

An ORM like JPA/Hibernate makes database access feel like working with plain objects. That is exactly the problem: a single innocent-looking loop can fire hundreds of SQL queries, and you won’t see it until production slows to a crawl.

This post covers the ORM traps that bite Java teams most often — starting with the notorious N+1 query problem — and the concrete techniques that fix them.

The N+1 Problem

Consider loading authors and printing their books:

Introduction

Python’s ORMs — SQLAlchemy and the Django ORM — are a joy to write and dangerously easy to make slow. A loop over a queryset that reads one related field can silently fire hundreds of queries. It looks fine in development and falls over on real data.

This post covers the ORM traps Python teams hit most, starting with the N+1 query problem, in both SQLAlchemy and Django.

The N+1 Problem

You load authors and print each author’s book count:

Introduction

Fixing N+1 queries (see the previous post) gets your Hibernate app down to a handful of queries per request. The next bottleneck is what each of those queries costs once your tables have millions of rows — and that is almost always a question of indexing.

An index turns “scan every row” into “look it up directly.” Get the index wrong — or skip it — and a query that took 2ms in development takes 4 seconds in production once real data volume shows up.

Introduction

Fixing N+1 queries with select_related/prefetch_related or selectinload (see the previous post) gets you down to a small, sane number of queries per request. The next bottleneck is what each query costs once the table has millions of rows — and that is almost always about indexing.

An index turns “scan every row” into “look it up directly.” Skip it, and a query that’s instant in development takes seconds once real data volume shows up in production.