Skip to content

[Server][DynamicAdmission] Preserve concurrency and queue state across runtime updates #331

Closed
Closed
[Server][DynamicAdmission] Preserve concurrency and queue state across runtime updates#331

Description

@SunSi12138
SunSi12138
opened on Aug 24, 2026

Parent: #264
Architecture dependency: #273
Previous slice: #329 / PR #330
Stack base: reviewed #330 exact head c1c68580e3337bf640c5c677cce829cae592f423.

This issue is the implementation manual for this slice. Re-check #330 before branching and use its latest reviewed exact head if it changes.

Goal

Add the first enabled -> enabled Dynamic Admission update path, limited to:

  • Global / Contract / Method concurrency add/remove/resize;
  • MaxQueuedCalls;
  • MaxQueuedBytes;
  • MaxQueueDelay;
  • QueueOneWayCalls.

The update must preserve real runtime state. A configuration change must not create a fresh concurrency budget, reset unchanged rate state, duplicate partition state, or split queue accounting.

This slice should expose one complete-candidate runtime update operation, conceptually:

server.UpdateAdmissionControl(options => { ... });

The callback defines the complete desired Admission configuration. Build/validate/reconcile happens off the request path; publication remains atomic and next-Request scoped.

This slice does not implement rate-parameter migration or partition-configuration migration. Those remain the next #264 slice.

Current code facts on #330

#330 already provides production Server-package control APIs:

server.EnableAdmissionControl(...);
server.DisableAdmissionControl();

and one shared publication/lifecycle path in SharpLinkServer.AdmissionProgram.cs.

The stable AdmissionStateKernel from #328 owns:

  • global queue count / byte accounting;
  • active permit accounting;
  • rule-state registry;
  • partition-state registry;
  • program retire/reclaim lifecycle;
  • shutdown sealing/drain.

Requests still capture one immutable AdmissionProgram generation exactly once, so old queued/active Requests continue using their captured policy snapshot after a new generation is published.

Current blocker: rule state identity includes mutable policy values

AdmissionRuleStateKey currently includes one combined AdmissionRuleStateDefinition containing:

ConcurrencyPermitLimit
Rate definition
QueueLimit

Therefore changing only concurrency or MaxQueuedCalls currently changes the state key and creates a fresh AdmissionRuleRuntime.

That is incorrect for this slice because:

N concurrency limit=100, active=80
update -> N+1 limit=50
fresh state active=0
N+1 admits 50 more
actual active=130

Also, changing MaxQueuedCalls must not reset an unchanged TokenBucket/FixedWindow/SlidingWindow state merely because BCL limiter queue capacity was part of the old combined state definition.

Current blocker: AdmissionRuleRuntime bundles concurrency + rate

A single AdmissionRuleRuntime currently owns both concurrency and rate limiter slots. This was sufficient for exact-compatible state reuse in #328, but it couples independent transition semantics:

  • concurrency is live-resizable in this slice;
  • rate parameters are immutable in this slice and must be preserved unchanged;
  • queue policy is program-level/shared accounting and must not define limiter-state identity.

The implementation must remove that accidental coupling rather than working around it with fresh controllers.

Public update semantics

Complete candidate

UpdateAdmissionControl takes a complete desired Admission configuration, like EnableAdmissionControl.

It is valid only while Admission is currently enabled.

Required control flow:

configure candidate
-> validate / resolve / compile
-> reconcile reusable runtime state
-> validate transition is in this slice's supported set
-> short lifecycle/publication critical section
-> verify current generation is still the expected source generation
-> publish N+1
-> retire N

User callbacks must execute outside publication/kernel locks.

Lost-update prevention

Two concurrent updates must not both derive from N and silently overwrite each other.

Candidate construction may be optimistic/off-lock, but publication must verify the source generation expected by the update is still current.

If current changed while the candidate was being built, fail predictably (recommended InvalidOperationException) and reclaim the losing candidate/state references. Do not silently rebase a caller's full replacement configuration.

Unsupported transitions

This slice must reject enabled -> enabled candidates that change out-of-scope state:

  • TokenBucket parameters;
  • FixedWindow parameters;
  • SlidingWindow parameters;
  • rate algorithm kind;
  • partition selector;
  • partition MaxPartitions;
  • partition IdleTimeout;
  • partition concurrency/rate configuration.

Unchanged rate and partition definitions must remain supported and must keep their existing runtime state.

A rejected transition is transactional: current publication/state remains unchanged and all candidate bindings are reclaimed.

Concurrency model

Stable concurrency state identity

Concurrency mutable state must be keyed by logical scope identity, not by its current numeric limit.

At minimum:

Global
Contract(contractId)
Method(contractId, methodId)

If a scope has concurrency in both N and N+1, N+1 must bind to the same stable concurrency state and reconcile its target limit.

If concurrency is newly added, create new state.
If concurrency is removed, N+1 no longer references it while old captured N Requests may continue; reclaim after old program/state users drain.

Do not make partition concurrency dynamic in this slice.

Required resize semantics

Increase:

limit=1
active=1
update -> limit=3
new requests may use the two newly available permits immediately

Shrink:

limit=3
active=3
update -> limit=1

Must mean:

  • keep all 3 existing holders;
  • admit no additional holder while active >= 1;
  • releases reduce active naturally;
  • capacity resumes only when active falls below the new limit;
  • no cancellation of active calls;
  • no transient fresh-generation permit budget.

The state must support acquire/release/resize races without underflow or overshoot.

Queue fairness during resize

Existing queued waiters must remain valid when concurrency changes.

Increase should wake/allow eligible waiters according to existing FIFO/oldest-first behavior.

Shrink must not cancel already queued waiters merely because the new limit is lower; they continue waiting under their captured queue/deadline semantics until capacity becomes legal, they time out/cancel, or Server Stop occurs.

Implementation shape

The exact type is not prescribed, but a dedicated stable ResizableConcurrencyState / limiter abstraction is expected if BCL ConcurrencyLimiter cannot safely express live resize.

If the implementation keeps the BCL limiter abstraction, it must still prove active count is preserved across resize. Replacing it with a fresh ConcurrencyLimiter is explicitly forbidden.

Split concurrency state from rate state

Refactor rule state so a concurrency change does not replace unchanged rate state.

Conceptually:

Rule binding
  -> optional stable ConcurrencyState
  -> optional stable RateState

Recommended identities:

ConcurrencyStateKey = scope identity
RateStateKey = scope identity + rate kind/definition generation

Exact class names are flexible.

Required invariant:

Token bucket under N has consumed quota
update only concurrency / queue policy
N+1 must reference the exact same rate state

No free burst, no window reset, no duplicate rate state.

This refactor should also remove MaxQueuedCalls from rate/concurrency state identity.

Queue policy model

Queue count/byte accounting is already server-wide in AdmissionStateKernel; preserve that single accounting domain.

Program generations continue to carry immutable queue policy snapshots.

MaxQueuedCalls

Increase:

old max=2, queued=2
update -> max=4
new N+1 requests may enqueue up to the new bound

Shrink:

old max=100, queued=80
update -> max=20

Must mean:

  • keep the existing 80 waiters;
  • do not cancel 60 of them;
  • new N+1 waiter is rejected while current queued count >= 20;
  • as old waiters leave, new queue admission resumes only when accounting falls below the new limit.

MaxQueuedBytes

Same semantics as count:

  • shrink does not evict old retained payloads;
  • new N+1 requests use the new bound against the one shared byte counter;
  • no double reservation/release;
  • old queued Request keeps its already reserved bytes until its normal terminal path.

MaxQueueDelay

Queue delay is captured when a Request enters the Admission queue.

Therefore:

A queued under N with 2s
update -> N+1 with 500ms
A keeps 2s
B queued under N+1 gets 500ms

Do not mutate timeout budget of existing waiters.

QueueOneWayCalls

This is per-program policy:

  • new OneWay Request captured under N+1 uses N+1 value;
  • already queued OneWay Request under N is not dropped when N+1 turns queueing off;
  • a Request captured under N cannot observe N+1's value later.

Decouple outer queue policy from inner limiter queue capacity

Today BCL concurrency/rate limiters receive MaxQueuedCalls as their internal QueueLimit. That makes queue policy part of limiter state construction and prevents safe queue-limit updates.

This slice must establish one authoritative queue bound: the stable kernel's outer queue reservation.

A queued Request may enter an underlying limiter async wait only after AdmissionStateKernel.TryReserveQueue(...) succeeds.

Refactor inner limiter queue capacity so changing MaxQueuedCalls does not require replacing concurrency/rate state.

Acceptable approaches include an internal limiter abstraction whose waiting is bounded by the kernel, or a fixed internal capacity that is provably bounded by the outer reservation. The implementation must prove there is no hidden unbounded waiter path.

Required invariant:

No waiter can become resident in an underlying limiter queue without already owning exactly one kernel queue reservation.

This must hold for concurrency, unchanged rate limiters, composite rules, cancellation, retry-to-next-slot, and partition paths.

Rule routing changes allowed in this slice

Global / Contract / Method concurrency may be added, removed, or resized through the complete candidate.

Examples allowed:

Global concurrency 100 -> 50
add Contract 123 concurrency=20
remove Method 123/456 concurrency

Rate definitions at every affected logical scope must remain unchanged unless the rate component is absent in both old and new configurations.

Do not allow a rule change to accidentally remove/add/replace a rate limiter in this slice.

If a scope had both concurrency + rate, changing/removing only concurrency must preserve the rate state and route semantics.

Partition boundary

Partition policy migration remains out of scope.

However queue-only or non-partition concurrency updates elsewhere must not duplicate or reset an unchanged partition pool.

The current AdmissionPartitionStateKey also incorporates combined rule definition and queue limit. Refactor identity enough that:

  • changing global queue policy does not replace an otherwise unchanged partition pool;
  • unchanged partition config remains exactly reused across N/N+1;
  • its internal unchanged concurrency/rate state remains valid.

Do not implement partition limit resize, selector replacement, MaxPartitions update, or IdleTimeout update here.

Candidate reconciliation must be transactional

A concurrency resize often needs to change stable mutable state before N+1 publication. That creates a critical transactional problem: if publication later loses a race, the current N must not accidentally observe a limit intended only for the losing candidate.

Do not mutate live shared concurrency limits irreversibly during speculative candidate build.

The implementation needs a prepare/commit model or equivalent, for example:

prepare candidate bindings + transition plan
publication lock verifies expected current generation
commit stable concurrency target changes
publish N+1
retire N

The commit/publication boundary must be linearizable.

If commit can fail, define rollback before publication or ensure all failure points occur before state mutation.

Required deterministic regression:

N limit=10
candidate A wants 5
candidate B wins with 20
A publication loses
live state must be 20, never left at 5

The request path must not take the control lock used to commit resize.

Update vs Enable/Disable/Stop races

All production writers must share the same lifecycle serialization semantics.

Required:

  • Update vs Update: exactly one expected-source publication wins;
  • Update vs Disable: linearizable final state; losing candidate reclaimed;
  • Update vs Stop: Stop seal prevents later publish and speculative transition has no effect;
  • Disable then Update: Update fails because current is disabled;
  • Enable then Update: update can proceed only from the actually current enabled generation;
  • no deadlock with generation retirement/reclamation or queued waiter completion.

Implementation sequence

1. Add deterministic failing tests first

Before replacing limiter internals, lock down:

  • concurrency shrink with active holders;
  • concurrency increase with queued waiter;
  • losing concurrent update cannot mutate live limit;
  • queue count shrink/grow;
  • queue byte shrink/grow;
  • MaxQueueDelay old/new generation semantics;
  • QueueOneWayCalls old/new generation semantics;
  • unchanged rate and partition state identity across update.

2. Split rule-state identity

Remove queue policy and mutable concurrency numeric target from identities that must survive updates.

Separate concurrency and rate state ownership enough that changing one cannot reset the other.

Keep static route lookup precompiled/immutable.

3. Introduce live-resizable concurrency state

Implement active-count-preserving resize and async wait behavior.

Audit exactly-once lease release and cancellation.

4. Decouple inner waiter capacity from dynamic outer queue policy

Make stable kernel queue count/bytes the authoritative admission queue bound.

Prove inner wait queues cannot exceed owned outer reservations.

5. Add production UpdateAdmissionControl

Extend the Server-package runtime control surface rather than adding an Abstractions -> Server dependency.

Use the same complete-candidate style as Enable.

Unsupported server implementations must continue to fail with NotSupportedException.

6. Add transition validation

Reject rate/partition changes in this slice before any live state mutation/publication.

7. Add atomic reconcile + publication

Use expected-source generation validation and a transactional commit plan.

Retire old program with existing #328 lifecycle after successful publication.

8. Documentation / benchmarks

Update doc/admission-control.md with:

  • UpdateAdmissionControl;
  • full replacement candidate semantics;
  • next-Request publication boundary;
  • concurrency shrink/increase semantics;
  • queue shrink semantics;
  • old waiter delay/OneWay capture semantics;
  • unsupported rate/partition update limitation.

Extend Admission benchmarks for steady-state after repeated concurrency/queue updates.

Required test matrix

Concurrency — Global

  • 1 -> 2 increase admits additional work without resetting active count;
  • 3 -> 1 shrink with 3 active admits no new holder until legal;
  • queued waiter survives shrink and eventually proceeds;
  • queued waiter is released promptly after increase;
  • repeated resize has no active-count underflow/overshoot.

Concurrency — Contract / Method

  • Contract concurrency resize preserves active state;
  • Method concurrency resize preserves active state;
  • add concurrency-only Contract rule;
  • remove concurrency-only Contract rule while old captured generation is active;
  • add/remove Method concurrency rule;
  • Global + Contract + Method composite continues to enforce all scopes correctly.

Mixed concurrency + unchanged rate

For TokenBucket, FixedWindow and SlidingWindow at least one deterministic preservation case each:

  • consume/advance rate state under N;
  • update concurrency or queue only;
  • N+1 uses the same rate state/history;
  • no free burst / fresh window;
  • old N queued/retained rate lease remains valid.

This is preservation coverage only; changing rate parameters remains rejected.

Queue count / bytes

  • MaxQueuedCalls increase;
  • MaxQueuedCalls shrink below current queued count without eviction;
  • MaxQueuedBytes increase;
  • MaxQueuedBytes shrink below current retained bytes without eviction;
  • new waiter uses N+1 limits while old waiter keeps reservation;
  • all queue counters return to zero after drain;
  • no underlying limiter waiter exists without kernel queue ownership.

Queue delay / OneWay

  • old waiter retains old MaxQueueDelay after update;
  • new waiter gets new delay;
  • existing queued OneWay survives QueueOneWayCalls=true -> false;
  • new OneWay under false does not queue;
  • false -> true permits new OneWay queueing;
  • two-way semantics remain unchanged.

Partition preservation

  • global queue policy update reuses unchanged partition pool;
  • non-partition concurrency update reuses unchanged partition pool;
  • active partition entries survive update;
  • consumed partition rate state is not reset;
  • candidate partition config change is rejected transactionally.

Transactionality / writer races

Use deterministic barriers:

  • invalid candidate leaves current program/state untouched;
  • rate-change candidate rejected with no live-state mutation;
  • partition-change candidate rejected with no live-state mutation;
  • concurrent Update vs Update: loser cannot leave its resize applied;
  • Update vs Disable;
  • Update vs Enable across disabled boundary;
  • Update vs Stop after candidate prepare;
  • losing candidate bindings/reconcile plan fully reclaimed.

Lifecycle / generation

  • old active Request under N completes after N+1 publish;
  • old queued Request under N continues with captured queue delay/OneWay policy;
  • N retires/reclaims after users drain;
  • repeated updates do not grow program/state registries unboundedly;
  • Stop drains current + retired generations exactly once.

ResourceGovernor regression

  • Admission update never changes ServerResourceGovernor ownership;
  • capacity-rejected compressed Request still has zero decompression/decoded rent;
  • pre-admission stream byte accounting remains exact across queued Request + update;
  • controlled rejection keeps the connection reusable.

Existing suite

Performance requirements

Disabled path must remain exactly as #330: no lock/refcount/no-op/allocation.

Enabled steady state after updates:

  • no request-path configuration lock;
  • no per-call lookup in kernel registries;
  • program route remains immutable/prebound;
  • concurrency acquire/release must not introduce avoidable allocation;
  • queue-limit dynamism must not add request-path work before a request actually needs to queue;
  • unchanged rate/partition immediate paths should not regress because concurrency became resizable.

Benchmarks should cover at minimum:

build-time enabled immediate permit
runtime enabled immediate permit
steady state after repeated concurrency resize
steady state after repeated queue-policy updates
immediate reject
queue/release

Report throughput/mean and B/op. Investigate stable >3–5% hot-path regressions rather than accepting them as the cost of dynamism.

Out of scope

Do not implement here:

  • TokenBucket parameter update;
  • FixedWindow parameter update;
  • SlidingWindow parameter update;
  • rate algorithm replacement;
  • partition selector replacement;
  • partition concurrency/rate resize;
  • dynamic MaxPartitions / IdleTimeout;
  • generalized rate/partition migration framework beyond what is necessary to preserve unchanged state;
  • dynamic-module Admission manifest/rule reconciliation redesign;
  • Hosting / IOptionsMonitor integration;
  • connection Admission dynamic configuration;
  • ResourceGovernor/decode/fair-scheduler changes.

If an out-of-scope transition is requested through the complete candidate, reject it transactionally instead of silently resetting state.

PR / validation procedure

  • Create the implementation branch from the latest reviewed [Server][DynamicAdmission] Add atomic runtime enable/disable control #330 head (c1c68580e3337bf640c5c677cce829cae592f423 at issue creation).
  • Stack the Draft PR on issue-329-runtime-admission-toggle / PR [Server][DynamicAdmission] Add atomic runtime enable/disable control #330.
  • Keep scope limited to this issue.
  • Use deterministic tests for resize/publication races; timing-luck stress is supplemental only.
  • Before leaving Draft, run full exact-head PR Quick.
  • PR body must record exact head SHA, PR Quick run ID, Unit/Generator/Load/Integration counts, AOT/package/load/chaos results, and explicit concurrency/queue state diagnostics.
  • Valid review findings require deterministic regression coverage and a fresh exact-head CI run.
  • Do not merge without explicit user authorization.

Definition of Done

  • public Server-package UpdateAdmissionControl (or equivalent) supports enabled -> enabled complete-candidate update;
  • update is transactional and expected-source/linearizable;
  • Global/Contract/Method concurrency add/remove/resize is supported;
  • concurrency shrink preserves active holders and never creates fresh budget;
  • concurrency increase exposes only the newly available capacity;
  • unchanged rate state survives concurrency/queue updates without reset/free burst;
  • MaxQueuedCalls and MaxQueuedBytes update against one stable queue accounting domain;
  • queue shrink never cancels existing waiter/retained payload;
  • existing waiter keeps captured MaxQueueDelay;
  • QueueOneWayCalls is next-Request/captured-program semantics;
  • outer kernel queue reservation is the authoritative bound for underlying limiter waits;
  • unchanged partition state is reused and partition changes are rejected in this slice;
  • losing/invalid update cannot mutate live stable state;
  • old N Requests remain generation-consistent and N reclaims safely;
  • Enable/Disable/Update/Stop writers are race-safe;
  • ResourceGovernor/feat(server): budget pre-admission stream buffers in ResourceGovernor #319 ownership is unchanged;
  • disabled and enabled immediate hot paths meet performance requirements;
  • documentation is updated;
  • full exact-head PR Quick is green;
  • PR remains unmerged until explicit authorization.

Follow-up slice

After this slice is reviewed, #264 continues with rate + partition state-preserving updates, including rate parameter migration/algorithm replacement and partition namespace/config transitions.

Refs #264, #273, #329, #330, #324, #328, #322, #323, #319.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions