SAI-210 knowledge check

How to use this assessment

Complete the questions without referring to the chapter text. Then review the guidance and update your lifecycle package where your answer exposes a gap.

This public knowledge check supports learning. It does not approve any named model or interpret its license.

Scenario questions

1. Acceptance before comparison

A team starts comparing candidate models before writing down what "acceptable" means for the workload. Explain why this ordering causes problems.

2. Hidden behavior change

Two deployments both use the same named base model, but one behaves noticeably differently from the other. List what else could differ between them.

3. License versus availability

A team treats the fact that a model's weights can be downloaded as evidence that the model may be used commercially. Explain the error.

4. Package as approval unit

An approval only references the model's weight file. Explain what else the approved package should bind together and why.

5. Evaluation expiry

A model passed evaluation eight months ago and nothing about its deployment has changed since. Under what conditions would that evaluation no longer be sufficient?

6. Aggregate scores hiding failures

A model's overall evaluation score is high, but it fails badly on a specific subgroup of cases. Explain why the aggregate score alone is misleading.

7. Retain versus authorize

A retired model's files are kept in storage "in case they're needed again." Explain why retention and authorization to use the model are different states.

8. Hidden invocation paths

A team removes a retired model's primary deployment and considers retirement complete. Name the kinds of paths that could still invoke it.

Answer guidance

1. Without acceptance criteria defined in advance — tasks, quality and safety expectations, constraints, prohibited uses — comparison becomes arbitrary, and candidates get judged against shifting or implicit expectations instead of the workload's actual requirements.
2. Tokenizer, configuration, quantization, prompt or harness, runtime, and hardware can all change served behavior while the named base model stays identical — evaluation and identity need to cover the full package, not just the model name.
3. Being able to download open weights says nothing about the license's usage and redistribution conditions. Permission must come from the recorded license text, source, and an internal legal decision, not from access alone.
4. The package should bind model and tokenizer identities to configuration, prompt or interface contract, runtime compatibility, evaluation results, license, provenance, policy, and limitations — approving only the weight file leaves the rest of what actually determines behavior unreviewed.
5. It stops being sufficient after any material change: new weights, tokenizer, quantization, prompt, runtime, hardware, retrieval, policy, deployment boundary, user population, data class, or a new security finding, incident, or detected drift.
6. A high aggregate score can conceal a subgroup or scenario where the model fails badly, because the average absorbs a small number of severe failures — evidence should report high-impact and subgroup failures explicitly, not just the overall number.
7. Retaining a model keeps its artifact and evidence available for investigation or reproducibility; authorizing it permits active use. A model can be legitimately retained while being explicitly prohibited from active use.
8. Routing aliases, caches, batch jobs, and other indirect or hidden endpoints can still invoke a retired model after its primary deployment is removed — retirement checks must confirm none of these paths remain live.

Completion rubric

Mark the lifecycle package complete only when:

• Purpose and acceptance criteria are explicit before any candidate is compared.
• Model, tokenizer, license, and provenance are traceable for every candidate considered.
• Approval is scoped to a specific package, not just a model name.
• The registry controls which versions can be consumed.
• Material changes trigger proportionate reassessment against the current baseline.
• Monitoring is connected to the properties the model was actually approved for.
• Rollback, revocation, and retirement are demonstrably workable, including hidden invocation paths.

Completion outcome

SAI-210 is complete when the learner can explain why a model name alone does not identify served behavior, and can produce the lifecycle decision package from the Model lifecycle workshop.