Skip to content

[bastion] Concurrent tunnel connections overwrite last_token; only last session is cleaned up #10137

Open

Description

@tikicoder

Describe the bug

With concurrent local TCP accepts on az network bastion tunnel, each connection requests a Bastion token via POST /api/tokens and stores it in a single instance field self.last_token. When active_connections reaches 0, cleanup() only DELETEs that last token.
Browsers / SOCKS clients that open multiple connections therefore create multiple Bastion active sessions (often with the same start time). After the tunnel stops, earlier sessions can remain visible in getActiveSessions / Portal until idle timeout.

Related command

az network bastion tunnel

Errors

no error but does not close all opened connections

Issue script & Debug output

the script does not error it just doesn't close all sessions

Expected behavior

When I close the tunnel I expect all sessions created to be closed.

Environment Summary

azure-cli 2.88.0

core 2.88.0
telemetry 1.1.0

Extensions:
account 0.2.5
ad 0.1.0
amg 3.0.0
azure-devops 1.0.6
bastion 1.4.3
databricks 1.3.2
datadog 3.0.0
front-door 2.3.0
resource-graph 2.1.1
ssh 2.0.9
stream-analytics 1.0.5

Dependencies:
msal 1.36.0
azure-mgmt-resource 24.0.0

Additional context

Summary

With concurrent local TCP accepts on az network bastion tunnel, each connection requests a Bastion token via POST /api/tokens and stores it in a single instance field self.last_token. When active_connections reaches 0, cleanup() only DELETEs that last token.

Browsers / SOCKS clients that open multiple connections therefore create multiple Bastion active sessions (often with the same start time). After the tunnel stops, earlier sessions can remain visible in getActiveSessions / Portal until idle timeout.

Related

Environment

  • azure-cli 2.88.0
  • bastion extension 1.4.3
  • Observed while using a local SOCKS5 proxy through az network bastion tunnel (multiple concurrent local accepts)

Code path (azext_bastion/tunnel.py)

  1. Each accept → _handle_client → _get_auth_token() overwrites self.last_token / self.node_id
  2. On client exit, when active_connections == 0, cleanup() deletes only self.last_token

So if connections A/B/C obtained tokens T1/T2/T3, only T3 is deleted when the last client leaves. T1/T2 are never deleted by this path.

Steps to reproduce

  1. Start az network bastion tunnel to a TCP/SOCKS resource port
  2. Open 2–3 concurrent local connections to the local tunnel port (or drive traffic through a browser SOCKS proxy)
  3. Check Bastion active sessions (getActiveSessions / Portal) — expect multiple sessions for the same user, same/near start time
  4. Stop the tunnel (Ctrl+C / kill the process)
  5. Re-check active sessions — earlier sessions may still appear

Expected

All tokens/sessions created by this tunnel process are cleaned up when their connections close and/or when the tunnel process exits.

Ask

Track tokens per connection (or a set of outstanding tokens) and DELETE each on close / process exit, instead of a single overwritten last_token.

Activity

  1. added
    bugThis issue requires a change to an existing behavior in the product in order to be resolved.
    on Jul 24, 2026
  2. yonzhan commented on Jul 24, 2026

    @yonzhan
    Collaborator

    Thank you for opening this issue, we will look into it.

  3. added
    questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
    customer-reportedIssues that are reported by GitHub users external to the Azure organization.
    on Jul 24, 2026
  4. removed
    questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
    Azure CLI TeamThe command of the issue is owned by Azure CLI team
    on Jul 24, 2026
  5. tikicoder commented on Aug 27, 2026

    @tikicoder
    Author

    Follow-up: per-connection Bastion session cost (latency) + cleanup interaction

    Same setup as the original report (az network bastion tunnel → local SOCKS5 → browser / tooling), bastion extension 1.4.3.

    What we measured

    For each local TCP accept on the tunnel port, the extension path still appears to:

    1. Acquire a token (MSAL)
    2. POST …/api/tokens
    3. Open a new wss://…/webtunnelv2/…
    4. On close: cleanup + DELETE /api/tokens (subject to the last_token overwrite bug above)

    End-to-end cost on our path is roughly ~650–800 ms per new SOCKS connect (Bastion path), while:

    • Direct TCP to the same kind of public endpoint is ~tens of ms
    • On the jump VM itself (SSH via the SOCKS tunnel, then local probe to Dante/SOCKS on the VM), connect latency is also ~tens of ms

    So the bottleneck is Bastion session/WebSocket setup per local flow, not the VM or the SOCKS daemon.

    Isolated Chrome (full SOCKS) can open ~50–58 concurrent connects to the local tunnel port on cold portal load. Concurrent accepts work (not fully serialized), but each still pays the per-session RTT. Scaling Bastion scale units does not remove that per-flow tax.

    How this relates to #10137 (cleanup)

    Under that concurrency, many tokens/sessions are created. Because only last_token is tracked for DELETE, most sessions are never cleaned up until idle timeout — which matches the original bug and makes long-lived browser/SOCKS use leave a large active-session footprint.

    Fixing per-connection token tracking / DELETE-on-close remains necessary.

    Additional ask (product / protocol)

    Even with correct cleanup, browsers and multi-connection SOCKS clients will keep paying hundreds of ms per flow. A durable fix would be for Bastion tunnel (service and/or CLI) to multiplex many local TCP flows over one (or few) WebSocket/session(s) instead of one Bastion session per accept.

    Client-side mitigations we are using (not a substitute for the above):

    • Cap concurrent proxy connections in Chrome (MaxConnectionsPerProxy) to reduce stampede
    • PAC allowlist so only private workloads (e.g. *privatelink*, storage/cosmos/databricks host patterns, RFC1918) go through SOCKS; portal/CDN/auth stay DIRECT

    Happy to provide redacted timing traces / diagnose output if useful (no tokens/JWTs).

  6. added a commit that references this issue on Oct 8, 2026
    535b9e5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Auto-AssignAuto assign by botNetwork - BastionService AttentionThis issue is responsible by Azure service team.act-quality-productivity-squadbugThis issue requires a change to an existing behavior in the product in order to be resolved.customer-reportedIssues that are reported by GitHub users external to the Azure organization.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions