Model evaluation, selection, and licensing
Compare model fitness, risk, licensing, operational constraints, and sovereignty consequences.
Define acceptance before comparison
State the tasks, languages, context distribution, quality and safety expectations, privacy constraints, latency, capacity, hardware, operating boundary, portability, and evidence needs. Include prohibited uses and failure cases.
Evaluate candidates on the same representative set. Record model and tokenizer identity, configuration, prompt or harness, runtime, hardware, dataset version, results, variance, and limitations. Aggregate scores should not hide high-impact or subgroup failures.
License and origin
Record the license text and version obtained with the artifact, source location, publisher claims, usage and redistribution conditions, required notices, restrictions, dependencies, and internal legal decision. Do not infer permission from the ability to download weights.
Selection record
Compare at least:
- Workload fitness and known failure patterns.
- Security, safety, privacy, and misuse exposure.
- Provenance and supplier transparency.
- Runtime and hardware feasibility.
- License and policy compatibility.
- Operational support, update path, portability, and exit.
- Evaluation and monitoring effort.
Record why the selected candidate is acceptable, where it is not, compensating controls, review expiry, and triggers for reconsideration.