Skip to content

fix(selfhost): Terraform firewall never opens Grafana's port for --profile observability #5818

Description

@JSONbored

Context

terraform/main.tf provisions a single Hetzner Cloud VPS for the self-host stack. Its
hcloud_firewall.gittensory resource opens exactly four inbound rules: TCP/22 (SSH), TCP/80 and TCP/443 +
UDP/443 (Caddy HTTP/HTTPS/QUIC), and TCP/8787 (direct app access, scoped to var.admin_ip_allowlist, with
a comment noting it's meant to be removed once behind Caddy).

docker-compose.yml's grafana service (used by --profile observability, the profile this same repo's
terraform/main.tf header comment tells the operator to run after provisioning — see its own top comment:
"After provisioning: SSH in, clone the repo, copy .env.example.env, and run docker compose up -d
(or docker compose --profile postgres --profile caddy up -d)") publishes Grafana directly to the host:

grafana:
  ports:
    - "3000:3000"

There is no firewall rule for port 3000 anywhere in terraform/main.tf, and unlike port 8787's rule there
is no var.admin_ip_allowlist-scoped opening for it either. An operator who provisions this VPS via
Terraform and then runs docker compose --profile observability up -d (a combination explicitly
documented as supported — see apps/loopover-ui/src/routes/docs.self-hosting-operations.tsx's "Resource
profiles" table, which lists --profile observability as one of the measured profile combinations) ends
up with Grafana bound to the host's public interface but with no path through the cloud firewall to reach
it — not even from their own IP.

For comparison, packages/loopover-miner/terraform/ (the sibling Terraform module for the AMS miner) ships
its own terraform/README.md explaining the deploy flow and explicitly notes in main.tf's own header
comment: "This is the CLI-worker profile — it exposes NO public endpoints by default (unlike the root
terraform/ module, which provisions the multi-tenant ORB server behind Caddy on 80/443)". The root
terraform/ module has no equivalent README.md at all (git ls-tree terraform/ shows only main.tf,
outputs.tf, variables.tf), and its own header/inline comments never mention Grafana, port 3000, or the
observability profile, so there is no documented path (SSH tunnel or otherwise) to reach Grafana on a
Terraform-provisioned host either.

Requirements

  • Add an inbound firewall rule for Grafana's port (3000) to hcloud_firewall.gittensory in
    terraform/main.tf, scoped to var.admin_ip_allowlist (matching the existing pattern used for port
    8787 — admin-only, not 0.0.0.0/0/::/0 like the public Caddy ports), OR — if a maintainer prefers
    Grafana stay unreachable by default and require an SSH tunnel — explicitly document that tradeoff with a
    comment in terraform/main.tf near the firewall block AND in a new terraform/README.md, including the
    SSH-tunnel command an operator would actually run (e.g.
    ssh -L 3000:localhost:3000 ubuntu@<server_ipv4>). Either resolution is acceptable; leaving the gap
    silently undocumented is not.
  • If adding the firewall rule: make it conditional/toggleable the same way other optional exposure is
    handled elsewhere in this repo's self-host tooling (config-as-code, not a business decision — this is a
    pure networking-parity fix, see the "safe to unleash" test in the repo's contributor-pipeline-gardening
    skill), e.g. a var.expose_grafana bool (default false, since observability is itself an opt-in
    profile) that gates the rule.
  • Add a short terraform/README.md (there is currently none) mirroring the shape of
    packages/loopover-miner/terraform/README.md: what gets provisioned, the deploy steps already described
    in main.tf's header comment, and now the Grafana access story (firewall rule or SSH tunnel, whichever
    this issue's implementation lands on).

Deliverables

  • terraform/main.tf's firewall either opens port 3000 (admin-allowlist-scoped, toggle-gated) or the
    SSH-tunnel-only tradeoff is explicitly documented in-file.
  • A new terraform/README.md documenting the provisioning flow and Grafana access path.
  • If a new var.expose_grafana (or equivalent) variable is added, it has a description and sane
    default in terraform/variables.tf, following the existing style of var.admin_ip_allowlist.

Test Coverage Requirements

This is Terraform HCL under terraform/** plus a new README — neither is under src/**, so Codecov's
patch-coverage gate does not apply. There is currently no terraform validate/plan step in this repo's
CI for the root terraform/ module (confirmed: no test references terraform/main.tf), so this issue is
config/docs-only and outside coverage.include; note that explicitly in the PR per this repo's own
template guidance rather than silently omitting a coverage section.

Expected Outcome

An operator who follows terraform/main.tf's own documented flow — provision the VPS, then run
docker compose --profile observability up -d — either reaches Grafana through a firewall rule scoped to
their own IP, or has an explicit, correct SSH-tunnel instruction instead of discovering the gap by getting
a timed-out connection with no explanation.

Links & Resources

  • terraform/main.tf (hcloud_firewall.gittensory, header comment referencing --profile observability
    indirectly via the general docker compose up -d flow)
  • terraform/variables.tf (var.admin_ip_allowlist, the existing admin-scoping pattern to follow)
  • docker-compose.yml (grafana service, ports: ["3000:3000"])
  • packages/loopover-miner/terraform/README.md and packages/loopover-miner/terraform/main.tf's header
    comment (the sibling module this issue's new README should mirror in shape/tone)
  • apps/loopover-ui/src/routes/docs.self-hosting-operations.tsx ("Resource profiles" table, confirms
    --profile observability is a documented, measured deployment combination)

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions