fix(egress): prevent DNS TTL expiry from killing long-lived TCP connections

[kinwin-ustc]

Summary

• DNS-learned allow_out_v2 entries now persist (expires_at_ns=0) instead of expiring after DNS TTL, preventing the eBPF datapath from RST'ing established TCP connections when TTL elapses.
• Increased CubeEgress proxy_read_timeout / proxy_send_timeout from 60s to 3600s to support long-lived HTTP connections (streaming, SSE, gRPC).
• The periodic reapDNSLearnedPolicies reaper is intentionally retained (currently a no-op since all entries have expires_at_ns=0) to preserve infrastructure for future TTL-based expiration if needed.

Background

When a sandbox has a long-lived TCP connection to a domain allowed via L7 network policy, the eBPF datapath (check_net_policy in from_cube) was killing the connection after DNS TTL expired — even though the domain was still in the allow list. This happened because:

1. dns_learn_response_ip() set expires_at_ns = now + DNS_TTL
2. After TTL expired, check_net_policy() treated the entry as absent
3. With allow_internet_access=false, traffic fell to deny_out (0.0.0.0/0) → TCP RST

Changes

File	Change
CubeNet/src/dns_response.h	Set expires_at_ns=0 in dns_learn_response_ip() — entries never expire
CubeEgress/nginx.conf	proxy_read_timeout / proxy_send_timeout: 60s → 3600s

Test plan

• Sandbox with L7 domain policy: establish long-lived HTTPS connection, wait > DNS TTL, verify data still flows
• Policy removal mid-connection: verify connection is terminated (replaceNetPolicy flushes the map)
• Verify DNS re-resolution still adds new IPs to allow_out_v2
• Verify sandbox teardown cleans up all entries
• Verify idle connections survive up to 1 hour without RST from CubeEgress proxy timeout
• Verify credential injection still works correctly on L7-proxied connections

[cubesandboxbot]

The comment on line 12-16 explains why reapDNSLearnedPolicies is no longer called here, but the function (and its helper reapDNSLearnedPoliciesForInnerMap at lines 31-89) remain defined with no callers. This is dead code that will bitrot: it compiles fine today even if the eBPF map iteration logic inside it breaks, because Go has no dead-code compilation errors.

Consider deleting both functions in this same PR, now that the sole call site has been removed. The comment at lines 11-16 already documents the design intent, so future readers won't wonder about the deletion.

[cubesandboxbot]

Consider adding a comment explaining the rationale for 3600s (e.g., "DNS-learned entries no longer expire on TTL, so connections may survive for the full hour"). Without context, a future maintainer might revert these to a lower default, reintroducing the connection-killing issue this PR fixes.

[cubesandboxbot]

The proxy_buffering on (line 86) with proxy_request_buffering on (line 87) means nginx buffers the full client request body before forwarding upstream, and buffers the full upstream response before sending to the client. Combined with the 3600s timeout, this can affect streaming/gRPC/SSE workloads where low-latency forwarding of partial data is expected. Consider whether proxy_request_buffering off; is needed for streaming use cases.

[cubesandboxbot]

Review: PR #763 — fix(egress): prevent DNS TTL expiry from killing long-lived TCP connections

Overall assessment

The PR correctly addresses the core bug: DNS-learned allow_out_v2 entries now have expires_at_ns=0 (never expire), preventing the eBPF datapath from RST'ing established TCP connections when DNS TTL elapses. The verdict is approve with comments — the approach is sound, but several documentation, robustness, and test-coverage gaps should be addressed before merging.

---

Documentation stale after this change

1. CubeNet/src/map.h:117-118 — Outdated invariant:

"A zero expires_at_ns means a static allow entry; a non-zero expires_at_ns means a temporary DNS-learned entry."

DNS-learned entries now also use expires_at_ns=0, so a zero value no longer implies "static policy entry." Update this to reflect that zero means "never-expire" regardless of origin.

2. CubeNet/src/dns_response.h:196 — Stale terminology:

"The path learns IPv4 A records into allow_out_v2 as temporary DNS-learned IP policy entries."

These are now persistent entries, not temporary. Suggest replacing "temporary" with "persistent (cleaned up on policy change or sandbox teardown)."

3. CubeEgress/nginx.conf:125-127, 194-196 — Lack of rationale:

The 60s→3600s timeout increase has no explanatory comment. A future maintainer has no context and may revert the values. Consider adding:

# Increased to 3600s to prevent timeout from closing long-lived connections;
# DNS-learned entries no longer have TTL-based expiration.

4. docs/guide/network-policy.md (lines 257, 396, 407, 410, 702) — The user-facing documentation references TTL-driven eviction of DNS-learned entries, which no longer applies. Update the docs to reflect the new persistent-entry behavior.

---

BPF map exhaustion risk (Medium)

dns_learn_response_ip now writes all entries with expires_at_ns=0. The reaper (reapDNSLearnedPolicies → netPolicyValueV2Expired) skips entries where ExpiresAtNS == 0, so DNS-learned entries accumulate until sandbox teardown or replaceNetPolicy. The per-sandbox inner LPM trie has a fixed capacity of 8192 entries (MAX_IP_RULE_ENTRIES). A sandbox that resolves many unique IPs (common with CDNs or cloud endpoints) can fill this map. When full, bpf_map_update_elem on line 129 uses BPF_ANY and its return value is not checked — the failure is silently swallowed and the sandbox stops learning new DNS-resolved IPs.

Consider:

• Checking the return value of bpf_map_update_elem and at least logging the failure.
• Incrementing a BPF map counter or metrics event when ENOSPC is hit.
• Adding a note in the function comment that the 8192-entry limit is now the only boundary (replacing TTL-based eviction).

---

Stale DNS-learned entries after policy updates (Medium)

applyNetPolicy (used by AddTAPDevice) does NOT flush the inner maps — only replaceNetPolicy (used by UpsertTAPDevice) does. Since DNS-learned entries are now permanent, removing a domain from the DNS allow list and then calling AddTAPDevice will not remove previously learned IPs from allow_out_v2. The sandbox retains egress access to those IPs until a full replaceNetPolicy or sandbox deletion. Verify that the AddTAPDevice → applyNetPolicy path either flushes DNS-learned entries or that this is documented as intentional behavior.

---

Test coverage gaps

The following areas lack test coverage directly relevant to this PR:

• dns_reaper.go: reapDNSLearnedPolicies, reapDNSLearnedPoliciesForInnerMap, and reapDNSQueryTrack have zero tests. The interaction between the reaper and permanent entries (ExpiresAtNS==0 → skip) is untested.
• populateAllowOutInnerMap (netpolicy.go:573-601): The Go-side mirror of BPF entry insertion has no tests for flag merging or the interaction between static and DNS-learned entries.
• Proxy timeout integration test: No automated test covers the 60s→3600s change. The PR test plan's "1-hour idle connection" scenario is manual only.

PR test plan note: The "policy removal mid-connection" scenario should explicitly verify the connection is terminated via policy-change-triggered flush (replaceNetPolicy), not via TTL expiry, since the reaper no longer removes entries.

---

Minor observations

• dns_response.h:166: The ttl local variable is read from the DNS RR header and passed to dns_learn_response_ip, where it is immediately cast to (void). Consider removing the ttl local and the parameter from dns_learn_response_ip if API callers permit it.
• dns_response.h:129: bpf_map_update_elem return value is unchecked. With the old TTL-based design, a full map was self-healing (entries would expire). With permanent entries, a full map is a hard stop for new learning — the error should be visible.
• Pre-existing: ngx.exit(ngx.ERROR) on nginx.conf:170 passes -1 as an HTTP status code; ngx.exit(500) or ngx.HTTP_INTERNAL_SERVER_ERROR would be correct.

---

Summary

Priority	Area	Issue
Medium	Docs	map.h:117-118, dns_response.h:196, nginx.conf missing comment, docs/ out of date
Medium	Correctness	Unchecked bpf_map_update_elem — silent ENOSPC with no recovery path
Medium	Correctness	applyNetPolicy doesn't flush DNS-learned entries on policy update
Low	Test coverage	dns_reaper.go, populateAllowOutInnerMap untested; no proxy timeout integration test
Low	Cleanup	Dead ttl read in dns_process_response_answer
Low	Preexisting	ngx.exit(ngx.ERROR) on nginx.conf:170

[chenhengqi]

This will block DNS reaper to do its job and the allow_out_v2 map will get full.