Skip to content

fix(temporal): bump role connectionLimit 50 → 200 - #134

Merged
mashkovd merged 2 commits into
mainfrom
fix/temporal-conn-limit
May 6, 2026
Merged

mashkovd merged 2 commits into
mainfrom
fix/temporal-conn-limit

Conversation

@mashkovd

@mashkovd mashkovd commented May 6, 2026

Copy link
Copy Markdown
Member

Schema setup hook fails with 'too many connections for role temporal'. 4 server pods × ~30 connections each saturate the original 50-conn ceiling. 200 leaves headroom.

mctl-deploy added 2 commits May 6, 2026 21:15
…obes

Three changes on top of the auto-generated values from
mctl_deploy_service onboard:

- dbSecret: pulls DB_USER/DB_PASSWORD from Vault
  teams/labs/kuptsi-app/database (provisioned earlier via
  mctl_provision_database) into a Secret consumed by envFrom.
  DATABASE_URL env composes the full DSN at startup so alembic +
  Temporal worker share the same shared-pg target.

- probes: kuptsi FastAPI exposes /health (src/app.py); base-service
  defaults aim at /healthz + /readyz which never become Ready.
  Aimed liveness + readiness at /health to match the app surface.

- extraContainers: Temporal worker as a sibling container on the
  same image. Single build, identical env (DB + Temporal address +
  user-managed Telegram/Logfire/Yandex secrets added via Backstage),
  only the entrypoint differs — `python -m src.temporal.worker`.
  Liveness probe is exec kill -0 since the worker has no HTTP.
  envFrom on the user-managed secret is `optional: true` so the pod
  starts before Backstage UI is used to add Telegram/Logfire/Yandex.
Schema setup hook crashed on 'too many connections for role
temporal' once all four server pods (frontend/history/matching/worker)
came online — each holds maxConns 20 (default) + 10 (visibility), so
steady-state usage already grazes 120 and admintools schema-setup
runs need a few extras on top. 200 gives comfortable headroom while
keeping a hard ceiling against runaway leaks.
@mashkovd

mashkovd commented May 6, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1b175ed7d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

# asyncpg DSN built from $(DB_USER)/$(DB_PASSWORD) injected by dbSecret below.
# The pydantic settings field name is `database_url` and the lifespan
# alembic step + Temporal worker both depend on it.
DATABASE_URL: "postgresql+asyncpg://$(DB_USER):$(DB_PASSWORD)@shared-pg-rw.platform-db.svc.cluster.local:5432/labs-kuptsi-app"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Provide DB env vars before interpolating DATABASE_URL

DATABASE_URL is now built from $(DB_USER)/$(DB_PASSWORD), but this values file still does not set envFrom for the main container. In base-service/templates/deployment.yaml, envFrom is rendered only from .Values.envFrom; setting dbSecret alone only creates the ExternalSecret and does not inject those vars into the app pod. With no DB_USER/DB_PASSWORD present, Kubernetes leaves unresolved references as literal strings, so the app/worker will attempt to connect with an invalid DSN and fail DB startup paths.

Useful? React with 👍 / 👎.

@mashkovd
mashkovd merged commit 86972fb into main May 6, 2026
@mashkovd
mashkovd deleted the fix/temporal-conn-limit branch May 6, 2026 19:27
mashkovd pushed a commit that referenced this pull request May 22, 2026
#91 Layer-2 merged via PR #132 (PR #124 closed unmerged; Layer 1 deferred
to issue #126). #94 (M4, scoped to Slices 1-2) merged via PR #125 with the
3 P3 nits cleared in PR #134.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant