Skip to content

Give each thread its own random number generator - #2783

Closed
neil-marcellini wants to merge 4 commits into
mainfrom
neil-srandom-thread-safety-test
Closed

neil-marcellini wants to merge 4 commits into
mainfrom
neil-srandom-thread-safety-test

Conversation

@neil-marcellini

@neil-marcellini neil-marcellini commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Details

(Neil's AI agent)

SRandom::_generator was a single process-global mt19937_64 with no mutex and no thread_local, and _distribution64 was likewise shared mutable state. Every draw advances that state, so concurrent draws from Bedrock's worker threads were a data race that handed the same "random" value to more than one thread.

This is not theoretical. Production Auth logs show four different threads on the same host receiving the identical reportID in the same microsecond:

06:44:33.359635  db1.sjc  socket12495613_cmd  req 9LxeT6  Generated random reportID: 8671345331629067
06:44:33.359636  db1.sjc  socket12494817_cmd  req ntOGuC  Generated random reportID: 8671345331629067
06:44:33.359636  db1.sjc  socket12495401_cmd  req cLSlQi  Generated random reportID: 8671345331629067
06:44:33.359637  db1.sjc  socket12495567_cmd  req p7Wt6T  Generated random reportID: 8671345331629067

The new test measures how bad it was: of 160,000 draws across 16 threads, only about 31,000 were unique before the fix. Roughly four in five concurrent draws were duplicates. That breaks the uniqueness assumption behind every randomly generated ID in Auth (reportIDs, reportActionIDs, transactionIDs), and read-then-insert guards like Transaction::generateID cannot save us, since two commands handed the same value both read "not present" and both insert.

The fix makes _generator and _distribution64 thread_local. The per-thread generator is seeded from random_device mixed with the thread id, because random_device can hand the same 32-bit value to two threads that seed at the same moment, which would leave them generating identical sequences. A mutex would also close the race but would serialize every ID draw.

One caveat on the test, which is written up in a comment alongside it: it fails every time on an arm64 dev machine, but it can pass on the x86-64 machines CI runs on, where the window between reading and advancing the generator's state is only a few nanoseconds wide. I confirmed that — CI came back green twice on the test-only commit with the bug still present. So it is a reliable local reproducer and a best-effort CI guard, and a green CI run on it is not evidence either way.

Fixed Issues

For https://github.com/Expensify/Expensify/issues/676823

Tests

(Neil's AI agent) Automated tests were added.

LibStuff::testRandomIsThreadSafe draws 10,000 values from SRandom::rand64() on each of 16 threads and asserts that all 160,000 are distinct. It fails 5 out of 5 runs without the fix (19–28% unique draws) and the fixture passes 39/39 with it.


Internal Testing Reminder: when changing bedrock, please compile auth against your new changes

(Neil's AI agent) Auth compiles and links cleanly against this change.

@neil-marcellini neil-marcellini changed the title Add a failing test for SRandom thread safety Give each thread its own random number generator Sep 11, 2026
@neil-marcellini

Copy link
Copy Markdown
Contributor Author

Closing as explained here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant