The OrcaRouter AI Gateway failover page opens with a sentence that is doing more work than it looks: a provider goes down, and no one notices [OURS]. That is a claim about where the failure is caught, and the phrase that supports it is pre-stream. A fallback that fires before the response begins streaming is invisible to the client. A fallback that fires after the first token has arrived is not a fallback at all — it is a truncated answer followed by an error, which is the thing users actually complain about.
This article is about what that timing guarantee buys, what it costs, and what it does not cover.
What the Chain Is
The mechanism is simple to state. You configure a fallback chain of up to five models, drawn from a catalogue of 200+ across 40+ providers. When an upstream rate-limits or returns a 5xx, the request is retried against a healthy model before the response has started streaming [OURS]. Health, weight and priority are tracked per channel, and unhealthy upstreams are taken out of rotation automatically. So the chain is not a static list tried blind — a channel that has been failing stops being a candidate.
Two properties of that design are worth naming.
It is a chain, not a pool. Order matters. The first model is the one you want; the rest exist so that a failure does not become an incident. A chain configured for cost will differ from one configured for quality, and both will differ from one configured for latency.
It is bounded at five. Five hops is enough to survive a multi-provider outage and few enough that a pathological loop is impossible. A request that fails all five returns an error, which is the correct behaviour — a retry chain that never gives up is a denial of service aimed at whoever you are calling.
What “Pre-Stream” Rules In and Out
The timing is the whole feature, so it is worth being precise about which failures it covers.
Covered: Connection failures, authentication errors, 5xx responses and rate-limit rejections that arrive before any body bytes. All of these happen before the client has seen anything, so a retry is indistinguishable from a slightly slower first token.
Partially covered: Responses that fail mid-stream. Once bytes are flowing, the client has seen a partial answer. A pre-stream chain cannot undo that, and the honest description is that the retry covers the next attempt rather than the current one.
Not covered: A model that returns a slow, wrong answer. Failover is about availability, not quality. A chain of five models all producing mediocre output will pass every health check.
That last boundary is the one most often confused. A failover chain is not a quality gate, and a gateway that routes on quality is a different product with a different mechanism.
The Receipt, Which Is the Part Teams Underestimate
Every retry is recorded: what failed, what served the request instead, and how long each attempt took [OURS]. That is a small feature with a large operational consequence.
Without it, a failover chain hides its own activity — the user sees a clean answer, and nobody learns that the primary model was down for two hours. With it, the same outage is a line in a log: the primary channel returned 503 for a window, the secondary served every request in that window, and the latency delta is measurable.
The practical value is that a failover chain becomes an observability feature rather than only an availability one. A team that reads its receipts learns which providers fail, when, and what the fallback costs in latency — which is information no provider’s status page will give you about another provider.
What It Costs, and Why Cost Is Not the Interesting Axis
Failover itself does not add a per-request fee on the free tier: the gateway’s pricing page states there is no per-request fee on Hacker or Team, with enterprise plans possibly carrying a flat per-million-token routing fee depending on volume [VENDOR]. So the cost of a failover chain is the cost of the fallback model, and only when it fires.
A request that succeeds on the primary costs what it always cost. A request that falls back to a cheaper model costs less; one that falls back to a dearer model costs more.
That means the interesting design question is not “can I afford failover” but “what should the chain be”. A chain that begins with a frontier model and falls back to a cheap one has a price that drops during an outage; a chain that begins cheap and falls back expensive has a price that spikes exactly when a provider is having its worst day — which is the worst possible time to discover a budget.
What a Chain Should Look Like
Five hops is enough room for a design, and the design that follows from the mechanics has a specific shape.
Hop one: the model you want. The default. If everything works, nothing else matters.
Hop two: the same model family from a different route. If the primary route is having an authentication or rate-limit problem, a second route to a comparable model preserves output quality. This is the hop that converts an outage into a latency blip.
Hop three: a nearby model with a similar profile. Different family, similar capability. Output quality drifts slightly; availability holds.
Hop four: a cheap, fast model. This is the hop that keeps the product responding when everything expensive is down. Its output will be worse, and the alternative is an error page.
Hop five: the smallest model in the catalogue. The last resort. At this point the goal is “return something coherent” rather than “return the best answer”.
The chain is also a budget statement. Every hop has a rate, and the sum of the rates weighted by expected failure frequency is the real cost of the design. On a healthy month the chain costs exactly what hop one costs; on a bad one it costs more and earns its keep.
What Failover Does Not Replace
Four things that a fallback chain is often assumed to cover and does not.
Idempotency. If a request was applied upstream and the response failed, retrying on another model applies it again. For a chat completion that is harmless; for anything with a side effect it is not, and no gateway can decide that for you.
Quality assurance. A chain of five healthy models producing poor output passes every check. Evaluation is a separate practice.
Rate-limit planning. A fallback that routes around a rate limit still hits the next provider’s rate limit. Concurrency limits are per account on the provider side, and a chain distributes load rather than removing it.
Provider-wide outages. A chain of five models from one provider is not a chain of five. The catalogue’s breadth is what makes a cross-provider chain meaningful, and choosing a chain is choosing which failure modes you have diversified against.
How to Think About a Gateway You Can Test
The useful property of a failover chain is that it can be verified, which is unusual for reliability features.
Point a client at the gateway, configure a two-hop chain, and deliberately break the first hop — an invalid key, a model name that does not exist, a provider that is refusing traffic. A working chain answers from the second hop with no visible interruption and a receipt that records both attempts. A broken chain errors on the first request.
That test takes minutes and it establishes something no status page can: that on the day a provider goes down, your product does not.