Introduction to model serving

Purpose of model serving

A model file does not define availability, latency, access, isolation, capacity, policy, or recovery. Model serving combines a runtime, an API contract, scheduling, hardware, routing, configuration, security, telemetry, and operational ownership into something that can actually be depended on. SAI-220 treats "the model is deployed" as the start of the design work, not its conclusion.

Start from a workload profile

The minimum inputs are:

• The model and context distribution, request and concurrency patterns, and output needs the workload actually produces.
• User groups, data classes, and the deployment pattern selected in SAI-120 — see Deployment boundaries and patterns.
• Latency, availability, and isolation objectives, stated as testable numbers, not adjectives like "fast" or "reliable."
• Failure behavior expectations and cost constraints.

A runtime or hardware decision made before this profile exists is a guess, however confident it sounds.

What makes model serving different from typical application serving

Conventional service architecture already covers APIs, routing, and capacity. Model serving adds:

• Accelerator scheduling and memory constraints that behave differently from typical CPU and memory service sizing.
• Load time and model-swap costs that can dominate latency in ways a stateless web service never experiences.
• A tighter coupling between model version, prompt, and runtime configuration — changing any one can change served behavior.
• Failure modes such as context overflow, queue saturation, or policy unavailability that need application-visible, non-misleading signals rather than a generic error.

The SAI-220 method

1. Separate model runtime, API contract, and workload responsibilities explicitly, even where one product combines them.
2. Design routing as a policy decision, not just a load-balancing mechanism.
3. Measure capacity under representative, declared conditions.
4. Plan degradation and recovery before a failure occurs, not during one.
5. Treat runtime, quantization, hardware, and routing changes as changes requiring evaluation.
6. Preserve local observability without exposing sensitive prompt or response content.

Module outputs

• A serving workload profile.
• A runtime and API-contract decision.
• A routing and hardware-capacity design.
• A reliability and degradation plan.
• Measured serving and release evidence.

Tool commands and hardware compatibility claims belong in versioned adapters, not in this conceptual method.