Repository navigation
[bastion] Concurrent tunnel connections overwrite last_token; only last session is cleaned up #10137
Description
Describe the bug
With concurrent local TCP accepts on az network bastion tunnel, each connection requests a Bastion token via POST /api/tokens and stores it in a single instance field self.last_token. When active_connections reaches 0, cleanup() only DELETEs that last token.
Browsers / SOCKS clients that open multiple connections therefore create multiple Bastion active sessions (often with the same start time). After the tunnel stops, earlier sessions can remain visible in getActiveSessions / Portal until idle timeout.
Related command
az network bastion tunnel
Errors
no error but does not close all opened connections
Issue script & Debug output
the script does not error it just doesn't close all sessions
Expected behavior
When I close the tunnel I expect all sessions created to be closed.
Environment Summary
azure-cli 2.88.0
core 2.88.0
telemetry 1.1.0
Extensions:
account 0.2.5
ad 0.1.0
amg 3.0.0
azure-devops 1.0.6
bastion 1.4.3
databricks 1.3.2
datadog 3.0.0
front-door 2.3.0
resource-graph 2.1.1
ssh 2.0.9
stream-analytics 1.0.5
Dependencies:
msal 1.36.0
azure-mgmt-resource 24.0.0
Additional context
Summary
With concurrent local TCP accepts on az network bastion tunnel, each connection requests a Bastion token via POST /api/tokens and stores it in a single instance field self.last_token. When active_connections reaches 0, cleanup() only DELETEs that last token.
Browsers / SOCKS clients that open multiple connections therefore create multiple Bastion active sessions (often with the same start time). After the tunnel stops, earlier sessions can remain visible in getActiveSessions / Portal until idle timeout.
Related
- Concurrent connections were added to fix bastion tunnel does not accept concurrent connections azure-cli#24600 (bastion extension ≥ 1.0.1)
- Adjacent:
az network bastion tunnelquietly drops connection azure-cli#28367 (tunnel drop / cleanup races)
Environment
- azure-cli 2.88.0
- bastion extension 1.4.3
- Observed while using a local SOCKS5 proxy through
az network bastion tunnel(multiple concurrent local accepts)
Code path (azext_bastion/tunnel.py)
- Each accept →
_handle_client→_get_auth_token()overwritesself.last_token/self.node_id - On client exit, when
active_connections == 0,cleanup()deletes onlyself.last_token
So if connections A/B/C obtained tokens T1/T2/T3, only T3 is deleted when the last client leaves. T1/T2 are never deleted by this path.
Steps to reproduce
- Start
az network bastion tunnelto a TCP/SOCKS resource port - Open 2–3 concurrent local connections to the local tunnel port (or drive traffic through a browser SOCKS proxy)
- Check Bastion active sessions (
getActiveSessions/ Portal) — expect multiple sessions for the same user, same/near start time - Stop the tunnel (
Ctrl+C/ kill the process) - Re-check active sessions — earlier sessions may still appear
Expected
All tokens/sessions created by this tunnel process are cleaned up when their connections close and/or when the tunnel process exits.
Ask
Track tokens per connection (or a set of outstanding tokens) and DELETE each on close / process exit, instead of a single overwritten last_token.
Activity
yonzhan commentedon Jul 24, 2026
Thank you for opening this issue, we will look into it.
tikicoder commentedon Aug 27, 2026
Follow-up: per-connection Bastion session cost (latency) + cleanup interaction
Same setup as the original report (
az network bastion tunnel→ local SOCKS5 → browser / tooling), bastion extension 1.4.3.What we measured
For each local TCP accept on the tunnel port, the extension path still appears to:
POST …/api/tokenswss://…/webtunnelv2/…DELETE /api/tokens(subject to thelast_tokenoverwrite bug above)End-to-end cost on our path is roughly ~650–800 ms per new SOCKS connect (Bastion path), while:
So the bottleneck is Bastion session/WebSocket setup per local flow, not the VM or the SOCKS daemon.
Isolated Chrome (full SOCKS) can open ~50–58 concurrent connects to the local tunnel port on cold portal load. Concurrent accepts work (not fully serialized), but each still pays the per-session RTT. Scaling Bastion scale units does not remove that per-flow tax.
How this relates to #10137 (cleanup)
Under that concurrency, many tokens/sessions are created. Because only
last_tokenis tracked for DELETE, most sessions are never cleaned up until idle timeout — which matches the original bug and makes long-lived browser/SOCKS use leave a large active-session footprint.Fixing per-connection token tracking / DELETE-on-close remains necessary.
Additional ask (product / protocol)
Even with correct cleanup, browsers and multi-connection SOCKS clients will keep paying hundreds of ms per flow. A durable fix would be for Bastion tunnel (service and/or CLI) to multiplex many local TCP flows over one (or few) WebSocket/session(s) instead of one Bastion session per accept.
Client-side mitigations we are using (not a substitute for the above):
MaxConnectionsPerProxy) to reduce stampede*privatelink*, storage/cosmos/databricks host patterns, RFC1918) go through SOCKS; portal/CDN/auth stay DIRECTHappy to provide redacted timing traces / diagnose output if useful (no tokens/JWTs).
[Bastion] Delete all concurrent tunnel session tokens on close