Request Layer - Ashutosh Jha

Request Layer

A request layer between the API gateway and an agent fleet: a 3-tier cache (exact, semantic, router-decision) with sub-20ms hits and an in-flight coalescer that collapses concurrent duplicate calls. Adds cost-aware ($/min) rate limiting and a 3-lane priority queue for stable overload handling, with per-request observability via Phoenix tracing. Won 1st place at the Nasiko Labs Buildthon.

<20mscache hit latency

3cache tiers

1stBuildthon

Problem

An agent fleet behind one gateway was paying the same upstream cost for duplicate work: identical requests hitting concurrently, semantic near-duplicates nobody deduped, and no way to shed load gracefully under overload. The result was hard failures.

Approach

Designed a 3-tier cache in front of upstream calls: exact match, semantic match, and router-decision reuse.
Built an in-flight coalescer that collapses concurrent duplicate calls into a single upstream request.
Added cost-aware ($/min) rate limiting and a 3-lane priority queue so overload degrades predictably instead of collapsing.
Instrumented every request with Phoenix tracing; made the whole layer opt-in and bit-for-bit identical when disabled.

Outcome

Sub-20ms cache hits on warm paths, materially lower upstream spend, and stable behavior at load. Won 1st place at the Nasiko Labs Buildthon.

Architecture

Request path: coalesce → exact cache → semantic cache → router-decision cache → upstream, with $/min limiting at admission.fig. 01

async def handle(req):
    key = fingerprint(req)
    if (hit := coalescer.join(key)):
        return hit                      # collapse dupes

    async with coalescer.lead(key):
        if (hit := await exact.get(key)):
            return hit                  # <20ms
        if (hit := await semantic.get(req)):
            return hit
        res = await limiter.run(
            lane=req.priority, upstream)
        await router_cache.put(key, res)
        return res

python

Highlights

Three-tier cache (exact, semantic, router-decision) serving sub-20ms hits.
In-flight coalescer collapses concurrent duplicate calls into a single upstream request.
Cost-aware ($/min) rate limiting with a 3-lane priority queue for stable overload handling.
Per-request observability through Phoenix tracing.
Opt-in, zero changes to existing services; bit-for-bit identical when disabled.
1st place at the Nasiko Labs Buildthon.

Stack

Python
FastAPI
Redis
Observability