Model-serving design workshop

Workshop objective

Create and test a runtime decision for a private assistant with interactive and batch traffic, restricted data, a fixed accelerator pool, and no automatic external fallback. Use synthetic workload figures and placeholder hardware inventories only.

Download SAI-220 runtime-decision-record templateAn ADR-style worksheet for comparing runtime and deployment options against measured capacity, hardware fit, and isolation.

Scenario

The workload serves an interactive chat surface during business hours and a nightly batch summarization job, over a fixed accelerator pool with no permission to fall back to an external endpoint. Reuse the deployment pattern selected in SAI-120; do not redefine the platform from scratch.

Step 1: Define the workload

Record the request distribution, quality needs, service objectives, and isolation boundary for both the interactive and batch traffic.

Exit check: interactive and batch traffic have separately stated objectives, not one blended number.

Step 2: Design the endpoint contract

Create the endpoint contract and the runtime responsibility model, naming who owns authentication, routing, policy inspection, and telemetry.

Exit check: the contract does not silently depend on a feature specific to one runtime.

Step 3: Compare runtime options

Using the downloaded template, compare at least two runtime or deployment options in an architecture decision record.

Exit check: the comparison is grounded in the workload profile from Step 1, not in vendor marketing claims.

Step 4: Design routing and fallback

Define routing inputs, approved destinations, deny behavior, health behavior, and any approved fallback — with no automatic silent fallback to an external endpoint.

Exit check: every route destination is either approved for this context or explicitly denied; none are undefined.

Step 5: Plan and run a capacity test

Create a representative capacity-test plan stating the declared model, hardware, and load conditions, and record its limitations.

Exit check: the capacity numbers state the exact conditions they were measured under.

Step 6: Design reliability and operations

Design indicators, alerts, degradation behavior, recovery targets, and incident runbooks for the serving endpoint.

Exit check: a process being alive is not treated as proof the service is healthy.

Step 7: Define the release gate

Define the release gate for model, runtime, configuration, and hardware changes, including what must be re-evaluated and what evidence is retained.

Exit check: every listed change type has a named evaluation depth, not just "test before deploying."

Workshop package

Deliver:

1. Workload profile with separated interactive and batch objectives.
2. Endpoint contract and responsibility model.
3. Runtime architecture decision record.
4. Route policy and approved-destination list.
5. Capacity-test plan and results or test design.
6. Reliability plan and incident runbook.
7. Release-gate definition for serving changes.

Passing work separates measured facts from assumptions and makes unsafe automatic fallback impossible by default. This workshop is an educational design exercise, not a certified capacity or hardware benchmark.