Skip to content

Hosted tenant containers never actually consume the central PostHog key #7876 injects #8626

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

#7876 ("Wire hosted control-plane + tenant containers to report to the central PostHog project")
was closed as delivered. Its Deliverable 1 reads: "Hosted control-plane and AMS tenant containers
initialize PostHog error tracking against the injected central key."

What actually landed (control-plane/src/container-driver.ts @8601, afb9d3cd6) only does the
control-plane half: createTenantContainer injects the fleet-wide key as an env var into every
tenant container's process env, under the name LOOPOVER_CENTRAL_POSTHOG_KEY
(CENTRAL_POSTHOG_KEY_ENV_VAR in control-plane/src/container-driver.ts:87):

export const CENTRAL_POSTHOG_KEY_ENV_VAR = "LOOPOVER_CENTRAL_POSTHOG_KEY";
...
if (config.centralPosthogKey) envVars[CENTRAL_POSTHOG_KEY_ENV_VAR] = config.centralPosthogKey;

The container-driver's own doc comment on ContainerDriverConfig.centralPosthogKey
(control-plane/src/container-driver.ts:44) explicitly frames this as making "the self-host image's
own error tracking (once its Phase-1 init lands) point at the loopover-owned project instead of
nothing." Phase-1 (#8287) has since landed: src/selfhost/posthog.ts's initPostHog() is the
self-host image's real init path, called from src/server.ts:336 as
await initPostHog(process.env). But initPostHog only ever reads POSTHOG_API_KEY:

// src/selfhost/posthog.ts:135-137
export async function initPostHog(env: NodeJS.ProcessEnv): Promise<boolean> {
  const apiKey = processEnvString(env, "POSTHOG_API_KEY");
  if (!apiKey) return false;

LOOPOVER_CENTRAL_POSTHOG_KEY is never read anywhere under src/ (confirmed by
grep -rn "LOOPOVER_CENTRAL_POSTHOG_KEY" --include="*.ts" . excluding control-plane/, which
returns zero matches outside control-plane/ and its own tests). There is no bootstrap/entrypoint
script inside the ORB self-host image that translates one env var name to the other either
(src/server.ts calls initPostHog(process.env) directly, with no pre-processing).

Contrast with the sibling env var the exact same commit injects the exact same way,
LOOPOVER_TENANT_SECRET_TOKEN (TENANT_SECRET_ENV_VAR, control-plane/src/container-driver.ts:81):
that one is consumed, by src/orb/broker-client.ts:105-111
(env.LOOPOVER_TENANT_SECRET_TOKEN used as the bearer token against ORB_BROKER_URL). The
central-PostHog-key half of the same commit has no equivalent consumer — it is dead wiring. A tenant
container started by the hosted control-plane today has LOOPOVER_CENTRAL_POSTHOG_KEY set in its
process env, but its PostHog error tracking is unconfigured and silently inert, exactly as if #7876
had never shipped, because the one env var initPostHog actually checks (POSTHOG_API_KEY) is never
set from it.

Requirements

  • src/selfhost/posthog.ts's initPostHog() (or its caller, src/server.ts) must resolve the
    self-host PostHog project key from POSTHOG_API_KEY first (operator override, unchanged, highest
    precedence) and fall back to LOOPOVER_CENTRAL_POSTHOG_KEY when POSTHOG_API_KEY is unset — the
    same precedence relationship LOOPOVER_TENANT_SECRET_TOKEN/LOOPOVER_PINNED_VERSION already have
    with their own operator-facing counterparts elsewhere in this codebase (a hosted-injected value is a
    fallback, never a silent override of an operator's own explicit config).
  • This must not change self-host's existing opt-in behavior for an operator who never runs inside a
    hosted tenant container: when neither POSTHOG_API_KEY nor LOOPOVER_CENTRAL_POSTHOG_KEY is set,
    initPostHog must still return false and stay a complete no-op, unchanged from today.
  • src/env.d.ts's self-host-only env-var documentation block (the comment block at
    src/selfhost/posthog.ts:15-20 describing which vars are self-host-only vs typed Env) must be
    updated to document LOOPOVER_CENTRAL_POSTHOG_KEY's existence and fallback precedence.

Deliverables

  • initPostHog() in src/selfhost/posthog.ts resolves its API key as
    processEnvString(env, "POSTHOG_API_KEY") ?? processEnvString(env, "LOOPOVER_CENTRAL_POSTHOG_KEY")
    (or equivalent, preserving POSTHOG_API_KEY's precedence), verified by a new test in
    test/unit/selfhost-posthog.test.ts (or the existing PostHog test file for this module) asserting
    that initPostHog({ LOOPOVER_CENTRAL_POSTHOG_KEY: "phc_central" }) returns true and activates the
    client using that key.
  • A new test asserting POSTHOG_API_KEY still wins when both POSTHOG_API_KEY and
    LOOPOVER_CENTRAL_POSTHOG_KEY are set simultaneously (operator override is never silently
    clobbered by the injected fleet-wide key).
  • A new test asserting initPostHog({}) (neither var set) still returns false — the no-op
    posture is unchanged.
  • control-plane/test/container-driver.test.ts and control-plane/test/driver-factory.test.ts
    (both already covering the injection side) are left passing unmodified — this issue is a
    consumer-side fix only, the control-plane injection side is already correct and out of scope.
  • src/selfhost/posthog.ts's header comment block (lines 15-20) is updated to name
    LOOPOVER_CENTRAL_POSTHOG_KEY as a fallback source alongside POSTHOG_API_KEY.

All five Deliverables are required in this one PR — there is no narrower scoping for this issue. A PR
that adds the fallback read but skips either the override-precedence test or the no-op-when-neither-set
test does not resolve this issue, because both are the exact regression guards that prevent this same
class of bug (an env var injected but never consumed) from recurring silently again.

Test Coverage Requirements

src/selfhost/posthog.ts is under src/**, measured by Codecov. Target 99%+ patch coverage,
branch-counted: both branches of the ?? fallback (POSTHOG_API_KEY present vs. only
LOOPOVER_CENTRAL_POSTHOG_KEY present vs. neither present) must each be hit by a dedicated test per the
Deliverables above — a single test that only exercises one branch does not satisfy this.

Expected Outcome

A tenant container started by the hosted control-plane, with CENTRAL_POSTHOG_KEY configured as a
control-plane secret, actually reports errors to the loopover-owned PostHog project via its own
src/selfhost/posthog.ts init path — the outcome #7876 was closed as having delivered, but does not
currently produce. An operator's own POSTHOG_API_KEY, when set, continues to take precedence
unchanged.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions