Caching in front of an agent fleet

Agent traffic breaks most of what you assume about caching. The same intent arrives phrased a dozen different ways. Identical requests stampede in parallel. A miss costs real money rather than a few milliseconds of CPU. Put a plain exact-match cache in front of that and it will barely earn its keep.

Building Request Layer, I ended up with three tiers. The first is an exact cache keyed on a normalized request. It is cheap, sub-millisecond, and it catches retries and refreshes. The second is a semantic cache over embeddings, so 'reset my password' and 'how do I change my password' land on the same answer when the policy allows it. The third caches router decisions, remembering which downstream path a request should take, so we do not pay to re-plan the same shape of work twice.

The in-flight coalescer surprised me most. Repeat requests spread over time are the case everyone designs for. The one that hurt was hundreds of identical requests arriving inside the same few hundred milliseconds. Collapsing those into a single upstream call and fanning the result back out did more for tail latency and spend than any individual cache tier.

None of this is worth much if you cannot see it. Every layer emits a span, so one trace tells you whether a response came from exact, semantic, router, or a cold upstream call, and what each one cost. Without that I would not have trusted the numbers, and I would have been right not to.

Starting again, I would build the tracing first and the caches second. Caches are easy to write and easy to get wrong in ways that never show up as errors. You cannot tune what you cannot measure.