The chat on this site looks simple. Pick a tier, send a message, read the answer. Behind that is a resilience system I built because depending on third-party LLM providers means depending on services that return 502s, rate-limit without warning, retire models without notice, and occasionally just stop.
Everything below is running in production in a single Firebase Function. Each pattern exists because something failed first.
The model chain
Visitors pick Fast, Balanced, or Thoughtful. Each tier maps to a free-tier model on OpenRouter, and the function holds those in an ordered chain: the Balanced model first, then Thoughtful, then Fast, then OpenRouter's managed free router as the last resort. If a request fails with a retryable code, the function tries the next model in the chain and logs why.
The managed router is deliberately last. Pinned models preserve the tier's character. The router guarantees an answer when the catalog changes underneath me, which it has, with three models retired at once. The chat kept answering. The only visible change was the model name under the replies.
The circuit breaker
Trying a failing model on every request is a good way to double your latency. The breaker tracks failures per model, opens after three failures within five minutes, and skips open models on later requests until the cooldown passes. The status endpoint exposes the breaker state, so the client can show a tier as unavailable instead of letting a visitor pick it and wait.
Two retry policies, and why they differ
This one was a real bug. The server has a list of provider error codes that mean "try the next model": 400, 404, 429, 502, 503, 504. The client has a list of codes worth retrying the whole request: 429, 502, 503, 504.
For a while the client copied the server's list verbatim. That meant a 404 from a retired model, which the server had already walked the entire chain to produce, got retried again by the client, consuming another rate-limit slot for a request that couldn't succeed. The lists are different because the layers are different. By the time the client sees a 400 or 404, every model has already been tried. The rule is now documented in the project's CLAUDE.md with the reasoning, so nobody unifies the lists again in the name of tidiness.
Cost safety
Only allowlisted models run. The allowlist records each model's provider and the date its free status was verified. A request for anything else throws a cost-safety error before the provider is called. This is the single mechanism that lets me leave a public chat running on a personal account without watching a dashboard.
Rate limiting that survives scaling
Ten requests per hour per visitor, keyed by an HMAC of the IP under a secret salt so raw addresses are never stored or logged. The window lives in Firestore so every function instance sees the same count, with an in-memory fallback if Firestore is unreachable. The salt is a Firebase Secret and the function refuses to serve in production if it isn't bound, on the reasoning that hashing with a public default salt over the small IPv4 space isn't meaningfully different from not hashing at all.
The limiter is scope-aware now. Device pairing for the Port companion has its own bucket, so a visitor pairing a device never consumes their chat quota, and the two features can't starve each other.
Local parity
Every production model has an Ollama stand-in with roughly the same character. When Ollama is detected in development, the function routes to it and the UI shows a local indicator. I test model changes against the stand-in before I deploy them, and the parity table is written into the docs as a requirement rather than a suggestion. Dev and prod that behave differently is how you find out about a retired model from a visitor.
What to log
Request ID on every response. Which model answered, and whether it was the requested one or a fallback. Circuit breaker transitions. Rate-limit decisions with the hashed key, never the address. Provider error bodies stay in the logs and never reach the browser, because a provider's error message is provider internals.
Degraded, not down
The principle under all of it: a visitor asking a question should get an answer from a smaller model before they get an error from a bigger one. Every pattern here serves that. The chain degrades quality before it degrades availability. The breaker protects latency. The allowlist protects cost. The limiter protects the whole thing from one bad actor.
None of this is exotic. It's the same resilience engineering any external dependency deserves, applied to a dependency that happens to be a language model.