Putting the model gateway inside your own process

#What a gateway is for

Once more than one part of a product calls a language model, the same requirements appear in each place. Route to a provider. Fall back when it is unavailable. Cache repeated requests. Redact personal data before it leaves. Enforce a budget per tenant. Record what was sent for audit.

Implementing that five times produces five inconsistent versions, and the one that matters during an incident is the one that was implemented least carefully.

#Why not a hosted one

Hosted gateways do this well and put a third party in the path of every prompt. That is a data governance question first: prompts contain customer data, and depending on the jurisdiction, sending them to an additional processor requires a legal basis and a contract [4].

It is also an availability question. The gateway becomes a dependency that can be down while both your service and the model provider are healthy, and you cannot fix it [5].

A library compiled into your process has neither property. You operate it, and you were already operating the process it lives in.

#What belongs in it

Routing with fallback. Strategies by cost, latency or priority, and the ability to fail over. Provider outages happen and a single hard-coded provider makes them your outage.

Caching with an honest key. Deterministic caching on the exact request is safe. Semantic caching, where a similar request returns a stored response, is useful and has a failure mode: two prompts that differ in one number are similar and want different answers. Semantic caching should be opt-in per route.

Guardrails on the way through. Redaction and injection detection at the gateway means every caller gets it, including the one added next month by someone who did not read the guidance [1].

Budgets per tenant. Usage-based cost with no per-tenant ceiling is an incident waiting for a loop in someone's integration.

An audit record. Model, version, prompt hash, token counts, decision. Regulatory frameworks increasingly expect traceability for consequential uses [3] [2], and reconstructing it later from application logs is not realistic.

#Compatibility as an adoption strategy

Exposing an interface that matches the most common provider API means existing code points at the gateway by changing a base URL. That single decision did more for adoption in my own projects than any feature, because it removed the migration from the decision.

#The part that is harder than it looks

Streaming. Four wire formats are in common use and they differ in how they frame reasoning steps, tool calls and multi-modal chunks. Supporting them all is more work than the routing, and caching a streamed response means recording and replaying it in a way that is byte-identical for the client.

I underestimated this by a wide margin.