Skip to content

fix(launch): raise the NOFILE soft limit to the hard cap at startup - #2125

Closed
angri450 wants to merge 1 commit into
TencentCloud:developfrom
angri450:fix/startup-nofile-soft-limit
Closed

angri450 wants to merge 1 commit into
TencentCloud:developfrom
angri450:fix/startup-nofile-soft-limit

Conversation

@angri450

Copy link
Copy Markdown

Problem

Issue #2060: on a 69-agent install, each agent holds a SQLite pool (~4 fds) for memory.sqlite, so ~562 fds are baseline against the default 1024 soft NOFILE cap. Slow growth over days tips the process over: [Errno 24] Too many open files ×92k, socket.accept() out of system resource ×92k, HTTP fully down until restart.

The issue's first expected-behavior item: "进程启动时把 soft limit 提到 hard(1048576)或至少高水位".

Fix

octop run (via run_foreground_blocking, the single foreground entry) now raises the soft RLIMIT_NOFILE toward the hard cap before serving:

  • POSIX only (import resource guarded; Windows returns immediately — the storm report is from Linux/Docker).
  • Best effort: if setrlimit is denied (hard-limit caps in containers, restricted daemons), log a warning and keep the current limit — boot never fails over this.
  • If the hard cap is RLIM_INFINITY, apply a conservative finite floor of max(soft, 65535) instead of infinity.

Not in this PR (as commented on the issue): SQLite connection reuse/pooling — that lives in octop-memory's per-agent runtime and deserves its own discussion there; and EMFILE backoff on the retry paths, which needs the specific no-backoff loop pinned down first.

Testing

New tests/unit/launch/test_nofile_limit.py (4 cases, stubbed resource module so they run on any platform):

  • soft raised to hard cap, setrlimit called with the new pair
  • no-op when soft already equals hard
  • RLIM_INFINITY hard cap → finite 65535 floor
  • denied setrlimit swallowed with a warning

ruff check / ruff format / mypy --strict on touched files: clean.

CHANGELOG

Entry added under [Unreleased] → 修复.

Fixes #2060

Multi-agent installs keep one SQLite pool per agent, so the common 1024
soft fd cap is the ceiling long before the hard cap matters (TencentCloud#2060): a
busy registry baseline sits near the soft cap and tips into an EMFILE
storm (92k 'Too many open files' errors in one morning on a 69-agent
install). Raise the soft limit toward the hard cap when the foreground
server boots; POSIX only, best effort — containers that deny setrlimit
log a warning and keep the current limit.

Fixes TencentCloud#2060
Copilot AI balanced review requested due to automatic review settings October 16, 2026 14:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@jubaoliang

Copy link
Copy Markdown
Collaborator

感谢贡献!本 PR 与 issue #2060(EMFILE)相关,但该问题已由 #2076(“fix: raise fd limits and coalesce accept resource errors”,2026-10-16 08:41 已合入 develop)完整修复,故关闭本 PR。

原因:

  1. fix: raise fd limits and coalesce accept resource errors #2076 新增 infra/utils/process_resources.py: ensure_nofile_limit(),并在 OctopServer.start()(infra/server.py:410)调用,把 soft 提升到 min(65535, hard)(含 POSIX 守卫 + best-effort),同时给 4 个 compose 文件加了 nofile: 65535。
  2. 相比本 PR 挂在 run_foreground_blocking,fix: raise fd limits and coalesce accept resource errors #2076 挂在 OctopServer.start() 覆盖面更广(CLI embedded 启动同样生效),并额外做了 accept 错误聚合。
  3. 本 PR 的唯一增量是「把 soft 提到完整的 hard cap」(fix: raise fd limits and coalesce accept resource errors #2076 封顶 65535,而 65535 是项目既定的 NOFILE_TARGET),属策略差异而非缺陷;其测试用例与 fix: raise fd limits and coalesce accept resource errors #2076 的 test_process_resources.py 重复。

建议:如果你确实希望放开到完整 hard cap,可以直接对 infra/utils/process_resources.py 提一个一行小改(把 min(NOFILE_TARGET, hard) 改为 hard),比保留整份重复 PR 更省事。再次感谢!

@jubaoliang jubaoliang closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants