AI Engineering

What actually limits an LLM application?

Rarely the model. In most production systems the binding constraints are retrieval quality, latency budget, and cost per request. All three are engineering problems with measurable answers, and none of which improve when you swap in a stronger model.

That is the working assumption behind this cluster. When a system gives bad answers, the first question is not “is the model good enough” but “did the model receive the right material, in a position it could use, within a budget the product can afford”.

Where do the real costs sit?

Three places, in roughly this order of impact:

• Tokens you did not need to send. Input is processed before the first output token appears, so an oversized prompt costs money and time-to-first-token on every request. Prompt caching (Anthropic, OpenAI) changes this dramatically for stable prefixes, and does nothing for traffic that does not reuse them.
• Retrieval you cannot measure. A pipeline without a labelled evaluation set is a pipeline where quality changes are invisible. Chunking strategy, reranking and passage ordering all move accuracy, and none of it is observable without measurement.
• Work done twice. Re-embedding unchanged documents, re-processing static context, and re-running deterministic steps inside a loop are the quiet majority of avoidable spend. Each one is small on its own, so none of them gets owned until somebody totals them up.

Why do these systems regress silently?

Because most of their failure modes are not exceptions. A cache prefix that stops matching, a retriever returning subtly worse passages, a prompt that grew past the model’s effective window. None of these throw. They surface as a slow drift in answer quality, or as a cost increase nobody attributes to the change that caused it.

The defence is instrumentation that treats these as first-class signals: cache hit ratio bucketed by deploy, retrieval quality against a fixed set, and token counts per request path. If a regression can only be detected by reading outputs, it will be detected late.

What does a healthy system look like in production?

Four signals, all cheap to instrument and all leading rather than lagging:

• Token counts per request path, split into cached reads, cache writes and uncached input. A change in the ratio points at a prefix that stopped matching long before the invoice does.
• Retrieval quality against a fixed set, re-run on every pipeline change. Without it, chunking and reranking changes are unfalsifiable.
• Time-to-first-token at the 95th percentile, measured at the tail. Input processing dominates it, and the tail is what users describe as “slow”.
• Answer quality bucketed by deploy version, so a regression names the change that caused it.

None of these require a vendor platform. All of them are numbers your own code already has access to. Systems drift because nobody wrote these down, which is a different problem from them being hard to obtain.

How should you read this cluster?

Every article here is written to be reproducible. Where a number appears, the method that produced it is described so you can re-run it against your own workload rather than take it on trust. Measurement beats intuition here, and that applies to what you read on this site as much as to anything you ship.