Activity

1. mickmis

   added a parent issue on Jul 7, 2026

   • Add flag to eliminate notification index contention #1541
2. mickmis commented on Jul 7, 2026

   

   mickmis

   on Jul 7, 2026

   Last edited by mickmis

   ContributorAuthor

   More actions

   In this comment I report various tests I've made in an attempt to understand better the issue and find a potential fix. Right now they are mostly inconclusive but reporting them can avoid doing twice the effort. Note that those tests were ran on my own laptop, meaning they can be compared among themselves, but hardly with other tests.

   DSS started locally with:

   INTER_USS_NETEM_CONF=delay 16.5ms 7.5ms 50% distribution paretonormal loss 0.025% 25%
   INTRA_USS_NETEM_CONF=delay 600us 40us 25% distribution normal loss 0.0005%
   NUM_NODES=3
   NUM_USS=3

   Locus start command:

   docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 -v .:/app/ interuss/monitoring-dev \
   uv run locust -f loadtest/locust_files/FlightsInSub.py -H http://dss.lb.localutm \
   -u 10 --cluster-count 1 --base-lat -34.928683 --base-lng 138.593795 \
   --area-radius 550 --max-flight-distance 100 --oi-duration 60 --uss-base-url http://dss.lb.localutm

   Tests

   Baseline without time-based index

   DSS setup:

   • v0.23.0-rc2
   • time-based notification index disabled (i.e. locks cells ahead of time)

   

   With time-based index

   DSS setup:

   • v0.23.0-rc2
   • time-based notification index enabled (i.e. NO cell locking)

   

   As expected, worse performances in this case.

   Replicas on all nodes

   DSS setup:

   • v0.23.0-rc2
   • time-based notification index disabled (i.e. locks cells ahead of time)
   • CockroachDB configured to have a replica store in 100% of the nodes

   

   No impact.

   Time-based index with alternative locking implementation

   DSS setup:

   • v0.23.0-rc2
   • time-based notification index enabled
   • BUT locking cells ahead of time anyway with alternative cell locking implementation:
     • avoid full scan by:
       • using index hint
       • filtering only by cell ID (and not by sub ID nor start/end)

   

   Mostly no impact. Maybe slight improvement on the 95th percentile, but could be statistically irrelevant.

   Time-based index with with alternative locking implementation BIS

   DSS setup:

   • v0.23.0-rc2
   • time-based notification index enabled
   • BUT locking cells ahead of time anyway with alternative cell locking implementation, building on the previous one:
     • avoid full scan by:
       • using index hint
       • filtering only by cell ID (and not by sub ID nor start/end)
     • split retrieving cells to lock from actually locking them
     • enable cockroachDB session setting optimizer_use_lock_op_for_serializable

   

   Slightly more interesting results, as the 50th percentile response time is significantly reduced, but no change for the 95th percentile (expected since there are failures). We may want to dig a bit more in that direction, starting by isolating the change that produced this improvement and assessing its impact on other test scenarios.

   Time-based index with original locking, disabling increment of notification indexes

   DSS setup:

   • v0.23.0-rc2
   • time-based notification index enabled
   • BUT locking cells ahead of time anyway with the orinal cell locking implementation
   • do not perform any subscription notification index increment

   

   Interestingly no impact on performances. Could this suggest the increment of the notification index is not the bottleneck here?

   Original time-based index, disabling increment of notification indexes

   DSS setup:

   • v0.23.0-rc2
   • time-based notification index enabled, no locking as in the original implementation
   • do not perform any subscription notification index increment

   

   Made this to see if that makes a different vs previous test. Not incrementing the notification index does not make a different either here.
3. barroco

   added

   P1High priorityHigh priority

   P2Normal priorityNormal priority

   and removed

   P2Normal priorityNormal priority

   on Jul 7, 2026
4. mickmis commented on Jul 30, 2026

   

   mickmis

   on Jul 30, 2026

   ContributorAuthor

   More actions

   Follow-up to report some limited tests about a finding we made, which is that for the locust test FLightsInSub.py, getting the client to reuse an existing subscription (i.e. to not create a new implicit subscription for every new flights) could show improvement, potentially significant ones.

   DSS setup:

   • v0.23.0-rc2
   • time-based notification index enabled (i.e. NO cell locking)

   Results on my local machine with 3 concurrent users, locust command:

   docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 -v .:/app/ interuss/monitoring-dev \
   uv run locust -f loadtest/locust_files/FlightsInSub.py -H http://dss.lb.localutm \
   -u 3 --cluster-count 1 --base-lat -34.928683 --base-lng 138.593795 \
   --area-radius 550 --max-flight-distance 100 --oi-duration 60 --uss-base-url http://dss.lb.localutm

   With new subscription for every flight, i.e. like the above, ~1.12 QPS:
   

   With using a single explicit subscription for all flights, ~2.63 QPS:
   

   I.e. more than double the QPS on my local machine. I doubt improvement would be as good in real life, notably because this test would be the equivalent of a single USS operating alone. But this still hints at some improvement just by reusing existing subscriptions.
   This is a result consistent with previous findings: the bottleneck is most likely in the round-trips that have to be made between CRDB instances, and the number of SQL queries made directly influences that. Not having to create implicit subscriptions reduce the number of SQL queries to be made, thus the round-trip, thus improve the QPS.
   Note that this is however something that needs to be implemented on the USS side.
5. barroco commented on Oct 29, 2026

   

   barroco

   on Oct 29, 2026

   Contributor

   More actions

   Closing this issue since investigation is done and parent issue has been resolved.
6. barroco

   closed this as completedon Oct 29, 2026