Vault shows "not configured" — sidecar/Job race crashes sidecar before KV v2 engine is created

[rowan-stein]

User Report

Vault integration is not working. Getting error in the UI: Vault not configured/unavailable. Showing graph-required secrets only. The issue occurs after the cluster has been running for some time. Vault is running and unsealed, but does not have the secret/ KV v2 engine mounted.

Specification

Root Cause: Sidecar/Job Race Conditions + Non-Idempotent Commands

The vault-auto-init sidecar and the vault-init-unseal Job both run the same script (vault_auto_init_script) concurrently. Both share the same Vault PVC (/vault/data). There are three TOCTOU race conditions where the loser crashes due to set -euo pipefail:

Race 1: vault operator init

Both the sidecar and Job check initialized == false and call vault operator init. The second caller gets exit code 2 ("Vault is already initialized") → set -e kills the script.

Race 2: vault token create -id="dev-root"

Both check if the dev-root token exists, both find it missing, both call vault token create. The second caller gets exit code 2 ("token already exists") → script crashes. Note: this happens INSIDE the if body where set -e still applies, even though ! VAULT_TOKEN=... vault token lookup uses negation.

Race 3: vault secrets enable -path=secret

Both check if secret/ mount exists, both find it missing, both call vault secrets enable. The second caller gets "path is already in use" → the explicit exit 1 in the error handler kills the script.

Failure Timeline ("Unsealed but no secret/ mount")

T+0s    Job starts, sidecar starts (both running vault_auto_init_script)
T+15s   Both install tooling (apk + vault binary download)
T+20s   Both see Vault uninitialized → Race 1: one inits, other crashes OR 
        Job wins init, sidecar picks up initialized state
T+22s   Both unseal Vault
T+23s   Both call ensure_dev_root_token → Race 2: one creates token, other crashes
T+23s   Sidecar crashes (set -e, exit code 2) → CrashLoopBackOff
T+23s   Job continues → ensure_kv_v2_secret_mount → creates secret/ engine → exits (EXIT_AFTER_UNSEAL=true)
        --- OR ---
T+23s   Job crashes, sidecar continues → creates secret/ engine → loops
        --- OR (worst case) ---
T+23s   Both crash at Race 2 → Job retries (backoff_limit=6), eventually succeeds and exits
T+30s   Sidecar in CrashLoopBackOff, must re-download tooling from internet
T+30s   If internet available: sidecar recovers after backoff, creates secret/ engine
T+30s   If internet unavailable: sidecar crash-loops indefinitely, secret/ engine NEVER created

The "unsealed but no mount" state occurs when:

1. The Job creates the dev-root token and secret/ engine, then exits (EXIT_AFTER_UNSEAL=true)
2. Later, Vault pod restarts → Vault re-seals
3. Sidecar restarts → tries to download tooling → fails (no internet) → CrashLoopBackOff
4. OR sidecar gets tooling → re-unseals → but crashes at Race 2/3 with the Job (if Job is still retrying)
5. Eventually Vault gets unsealed but secret/ engine is not re-created because the sidecar is in CrashLoopBackOff

Required Fixes

Fix 1: Use Vault image for sidecar and Job (eliminate internet dependency)

Replace alpine:3.19.1 with public.ecr.aws/hashicorp/vault:1.17.2 for both containers. Remove the Vault CLI download from ensure_tooling(). This eliminates the internet dependency on restarts.

In vault_values local, extraContainers section:

• Change image from "public.ecr.aws/docker/library/alpine:3.19.1" to "public.ecr.aws/hashicorp/vault:1.17.2"
• Remove the VAULT_VERSION env var

In kubernetes_job_v1.vault_init_unseal resource:

• Change image from "public.ecr.aws/docker/library/alpine:3.19.1" to "public.ecr.aws/hashicorp/vault:1.17.2"

In vault_auto_init_script:

• Remove the VAULT_VERSION variable
• In ensure_tooling(): remove the entire Vault CLI download block, remove unzip from package list

Fix 2: Make race-prone commands idempotent (eliminate crashes)

All three race-prone operations must handle "already exists" gracefully instead of crashing:

ensure_initialized() — after vault operator init fails, re-check if Vault is now initialized:

log "initializing vault"
if ! init_json="$(vault operator init -key-shares=1 -key-threshold=1 -format=json 2>&1)"; then
  # Another process may have initialized first — check again
  if vault status -format=json 2>/dev/null | jq -e '.initialized == true' >/dev/null 2>&1; then
    log "vault was initialized by another process; loading artifacts"
    load_artifacts
    return
  fi
  log "ERROR: vault operator init failed: $init_json"
  exit 1
fi

ensure_dev_root_token() — after vault token create fails, verify the token now exists:

if ! VAULT_TOKEN="$root_token" vault token lookup "$DEV_ROOT_TOKEN" >/dev/null 2>&1; then
  log "creating dev-root token"
  if ! VAULT_TOKEN="$root_token" vault token create -id="$DEV_ROOT_TOKEN" -policy=root >/dev/null 2>&1; then
    # Race: another process created it first
    if VAULT_TOKEN="$root_token" vault token lookup "$DEV_ROOT_TOKEN" >/dev/null 2>&1; then
      log "dev-root token created by another process"
    else
      log "ERROR: failed to create dev-root token"
      exit 1
    fi
  fi
fi

ensure_kv_v2_secret_mount() — after vault secrets enable fails, re-check if the mount exists:

log "enabling kv v2 secrets engine at path secret/"
if ! VAULT_TOKEN="$root_token" vault secrets enable -path=secret -version=2 kv >/dev/null 2>&1; then
  # Race: another process may have created it
  recheck="$(VAULT_TOKEN="$root_token" vault secrets list -format=json 2>/dev/null || true)"
  if printf '%s' "$recheck" | jq -e '."secret/" | select(.type == "kv" and ((.options.version // "") | tostring) == "2")' >/dev/null 2>&1; then
    log "secret/ engine created by another process"
  else
    log "ERROR: failed to enable kv v2 secrets engine at secret/"
    exit 1
  fi
fi

Fix 3: Remove redundant vault-init-unseal Job

The sidecar already handles init, unseal, dev-root token, and KV v2 engine creation. The Job duplicates this work and is the source of all race conditions. Removing it eliminates the races entirely.

Remove the entire kubernetes_job_v1.vault_init_unseal resource and its kubernetes_config_map_v1.vault_init_unseal config map.

Files to Change

• stacks/platform/main.tf