Telemetry and AI quality signals

Map the request lifecycle

Correlate admission, identity, policy, routing, retrieval, inference, tools, response, user feedback, and evidence without placing all raw content in one log.

Signal catalog

For each signal, state definition, source, dimensions, unit, sampling, collection delay, retention, access, quality limits, owner, and decision it supports. Useful groups include:

• Traffic, errors, latency, saturation, queueing, timeouts, and cancellations.
• Model and runtime identity, context/output distribution, loading, memory, and accelerator state.
• Retrieval support, citation coverage, no-answer, permission decisions, and stale-content indicators.
• Policy allows, denies, redactions, approvals, routes, egress, and tool outcomes.
• Offline evaluation, production sampling, human correction, escalation, and accepted completion.

Quality and privacy

Do not treat user approval clicks or model-based grader scores as ground truth without calibration. Track disagreement and sampling bias. Prefer structured outcome and reason codes, protected references, and bounded diagnostic capture over default prompt/response logging.

Test signal loss, duplicate events, clock drift, missing versions, cardinality explosions, and telemetry backpressure. Observability failure can itself be a control event.