Confidence / uncertainty display

Problem Context

• The product uses model output, rules, retrieval, extraction, classification, ranking, risk scoring, or generated text to support a user decision.
• The system can estimate reliability, uncertainty, missing input, conflicting signals, distribution shift, source weakness, model freshness, or calibration limits.
• The user's action can affect customers, money, access, legal status, safety, operations, compliance, or public communication.
• A confidence display may appear inside chat, work queues, review panels, recommendation cards, form extraction, decision support, incident triage, or content publishing flows.
• A high confidence score is not the same as source proof, policy permission, legal correctness, or a severe consequence warning.

Selection Rules

• Choose confidence / uncertainty display when users need to judge prediction reliability before acting on AI or automation output.
• Use source grounding display when the task is answer-wide source coverage, searched corpora, used evidence, and unsupported claims.
• Use citation display when the task is inspecting one claim-level source, quote, metadata, or source permission state.
• Use meter when the value is a read-only bounded scalar gauge and not an AI reliability state with calibration, threshold, and review implications.
• Use warning text when the message is a known severe consequence before an action rather than a reliability estimate.
• Use recommended next action when the system is proposing the action itself; confidence can appear inside it but should not replace reason, consequence, and review controls.
• Use query correction or adaptive defaults when confidence only gates those specific workflows.
• Expose the threshold that changes behavior, such as auto-apply, review required, source verification required, specialist review, or no automation.
• Do not show a precise percentage unless the number is calibrated, current, scoped to this task, and meaningful to the user's decision.
• Do not let confidence imply correctness, source support, policy approval, permission, or safety beyond the measured reliability dimension.

Required States

• High confidence state with calibration scope, reason, and whether direct apply is allowed.
• Medium confidence state with review recommendation, uncertainty reason, and threshold context.
• Low confidence state with apply gated and a review, verify, collect more input, or fallback action.
• Insufficient evidence state when the model cannot assess the input because required signals are absent.
• Conflicting signals state when evidence or model features disagree and need human judgment.
• Out-of-distribution or unfamiliar input state when the model sees content outside its trained or monitored operating range.
• Stale model, stale data, stale source, or threshold changed state that invalidates previous confidence.
• Calibration unavailable or uncalibrated score state that avoids fake precision.
• Review required, explain confidence, blocked apply, and mobile compact panel states.

Interaction Contract

• The display names the task, prediction, source of the confidence estimate, calibration scope, and last calibration or update time when available.
• Confidence labels, bands, and numeric values are tied to threshold rules that determine apply, review, escalation, verification, or fallback behavior.
• Users can open an explanation that identifies the main reliability drivers without exposing sensitive training data or protected signals.
• Low confidence, insufficient evidence, conflicting signals, out-of-distribution input, stale model, and calibration unavailable states change available actions before the user commits.
• Regeneration, edited input, changed source scope, model updates, threshold changes, and new evidence invalidate or recompute the visible confidence state.
• Copy, apply, approve, publish, send, deny, or export actions carry the current uncertainty state or are gated when the threshold is not met.
• Users can report incorrect confidence, override with reason where policy allows, or route the item to a reviewer without losing context.

Implementation Checklist

• Define the prediction, confidence source, calibration method, operating scope, threshold rules, freshness policy, invalidation events, and user actions before rendering the display.
• Model high, medium, low, insufficient evidence, conflict, out-of-distribution, stale, uncalibrated, threshold changed, review required, and gated-action states as structured data.
• Separate model confidence from source grounding, claim citations, policy rules, permission checks, warning text, task status, and user feedback.
• Show the confidence label, calibrated range or caveat, threshold, reason, freshness, and recommended next action close to the AI output.
• Gate high-impact actions below threshold with review, verification, specialist escalation, or fallback, and re-check thresholds immediately before commit.
• Record overrides, reviewer decisions, user feedback, confidence state, threshold version, model version, and input version for audit where outcomes matter.
• Test fake precision, missing calibration, stale model, threshold changes, conflicting signals, no evidence, mobile wrapping, keyboard flow, screen-reader output, and high-zoom layouts.

Common Generated-UI Mistakes

• Showing a fake percent or exact decimal for an uncalibrated model score.
• Using color-only confidence badges without text, threshold, reason, or action guidance.
• Treating high confidence as proof, source support, legal approval, or permission.
• Hiding low confidence, insufficient evidence, conflicting signals, or out-of-distribution states behind an action menu.
• Letting stale confidence persist after source, input, model, threshold, or policy changes.
• Using confidence display to avoid exposing source grounding, citations, warnings, or review workflows.
• Making low-confidence automation look like the normal primary path because the review option is buried.

Critique Questions

• Is the displayed confidence calibrated for this exact task, population, model version, and threshold?
• Can users tell what action is safe at high confidence, medium confidence, low confidence, and insufficient evidence?
• What invalidates the displayed uncertainty after edits, regeneration, source changes, model updates, or threshold changes?
• Does the UI separate model reliability from source grounding, citation evidence, policy approval, permissions, and severe consequence warnings?
• Can users open an explanation, request review, correct feedback, or override with reason where policy permits?
• Does the mobile layout keep uncertainty, threshold, and gated action visible before users commit?