Routing, capacity, and hardware
Design routing and hardware profiles from measured workload envelopes, isolation needs, and failure behavior.
Routing is a policy decision
A router may select a destination by approved workload, model capability, data classification, latency, region, tenant, health, capacity, or cost, and by fallback state. Record which inputs are authoritative for that decision, when a route is denied outright, and which destinations are allowed for each context — a router without an explicit deny path will eventually route somewhere nobody approved.
Do not silently route to an external or less-controlled endpoint when local capacity is exhausted. A fallback destination changes the request's boundary and needs independent approval and evaluation, not an automatic failover rule buried in configuration.
Capacity profile
Measure with a declared model, tokenizer, precision, runtime, hardware, context distribution, input and output lengths, concurrency, batching, and quality settings — a throughput number without these conditions cannot be reused for planning. Capture throughput, time to first output, completion latency percentiles, queue time, error rate, memory headroom, saturation, and stability over a sustained period, not just a short burst.
Hardware decision
Consider accelerator availability, memory, interconnect, CPU and memory support, storage load time, power and cooling, isolation, firmware and driver lifecycle, supply constraints, and recovery. Higher peak throughput may not improve the workload if queueing, context size, or memory pressure dominate actual behavior — buying faster hardware does not fix a design that is bottlenecked elsewhere.
Plan headroom for failures, maintenance, model loading, and demand variance, and link capacity assumptions directly to monitoring and admission control so a violated assumption is visible rather than discovered during an incident. See Capacity, reliability, and recovery for the shared reliability pattern this connects to.
Record every capacity assumption in the runtime architecture decision record so a future hardware or model change is evaluated against the same declared baseline rather than against an untested new set of assumptions.