feat(observability): standardize error-recovery telemetry #3182

Priority

Wave 3 — incremental observability under #2831. Instrument stable behavior as it lands; this is not a #2750 completion gate.

Problem

Gym has OpenTelemetry foundations, but retry, outcome, masking, cleanup, cancellation, cohort, and stall signals do not use one low-cardinality convention. Operators cannot reliably quantify recovery depth, delay, unresolved sessions, masked results, or stalled work.

Essential scope

• Common metric names and span attributes for stable error-recovery facts.
• Retry attempts, cumulative delay, exhaustion, and minimal stage/delivery context.
• Completed, failed, omitted, unknown, and masked counts using registered low-cardinality kinds.
• Session release outcome and latency, expiry, abandonment, and unresolved cleanup.
• Cancellation, cohort timeout, and stalled-work age where the owning producer exposes them.
• Correlation identifiers in logs and traces, never as metric labels.
• A minimum structured-log contract when OpenTelemetry is disabled.

Scope boundaries

• feat(core): define a shared failure-kind registry #3179 owns failure names.
• feat(core): bound HTTP retries and make request replay explicit #3159/fix(model): bound provider-response retries #2845 own retry behavior.
• feat(core): define typed rollout outcomes and recovery #2135 owns final rollout dispositions.
• feat(core): define completed verifier-result failure semantics #2608 owns masking semantics.
• feat(core): define resources-session identity and close protocol #2609/feat(resources): add session TTL and lifecycle adoption #3037 own session cleanup.
• fix(genrm): bound incomplete cohorts and preserve cohort accounting #3181 owns cohort behavior.
• feat(core): own startup, shutdown, and cancellation #3160 owns local lifecycle.
• feat: live visibility into in-progress rollouts #2859 owns live presentation.

Basic logs necessary to operate a producer ship with that producer. This issue standardizes exported telemetry and does not delay correctness fixes.

Dependencies

No all-or-nothing dependency on every producer. Naming work and instrumentation for already-stable events proceed immediately; producer-specific fields integrate as each contract stabilizes.

Acceptance criteria

• One documented low-cardinality convention covers stable retry, outcome, masking, cleanup, cancellation, cohort, and stall facts.
• Unique identifiers appear only in logs and traces, not metric labels.
• Structured logs preserve useful recovery facts when telemetry is disabled.
• Tests assert exported names, bounded attributes, cardinality constraints, and disabled-observability behavior.
• Instrumentation lands incrementally without changing runtime policy.