perf: use a random pool

[ronag]

In production randomFillSync is taking 8-9% of our cpu time. This tries to reduce the overhead by pooling random data.

[sdrsdr]

I like the idea in general, but I'm not a maintainer of the project. I hope your PR get merged :)

[sdrsdr]

Much simple problem than I thought initially! I really do hope this gets merged!

[sdrsdr]

Fingers crossed this gets trough :)

[lpinca]

I'm not convinced this is much better than the status quo. It would only take 8 WebSocket each sending a single message to empty the pool and there is a new Buffer allocation every time.

[ronag]

This is just the fallback path. We shouldn't hit this often, if at all.

[ronag]

I think you are misunderstanding something here...?

[lpinca]

I think that if there are a lot of sockets all of them sending data at the same time synchronously, then this does not change much. I would like to see a "before" and "after" comparison. Do you have a flamegraph?

[lpinca]

Also from my experience generating random data is a nothing compared to the actual masking of the data in terms of CPU time.

[sdrsdr]

It will be best to skip masking for TLS websockets..

[lpinca]

It will be best to skip masking for TLS websockets

No sure if it is spec compliant. What about proxies?

[ronag]

Also from my experience generating random data is a nothing compared to the actual masking of the data in terms of CPU time.

From my profiles this is not the case.

[ronag]

I think that if there are a lot of sockets all of them sending data at the same time synchronously, then this does not change much.

Why? We generate the random data asynchronously/parallel off thread so most of the overhead is off the main thread.

[ronag]

Do you have any existing good benchmark we can use for before/after?

[lpinca]

From my profiles this is not the case.

Is it possible to share the code/data? My tests were based on crypto.randomBytes() (sync) but I would be surprised if crypto.randomFillSync() is actually worse.

Why? We generate the random data asynchronously/parallel off thread so most of the overhead is off the main thread.

This is not the case if we are near the end or already emptied the pool and we need data now.

Do you have any existing good benchmark we can use for before/after?

No, but a simple echo server with a bunch of clients sending data should work.

[ronag]

This is not the case if we are near the end or already emptied the pool and we need data now.

We dispatch the next parallel task before the pool has emptied. RANDOM_POOL_REFRESH

[ronag]

That's not much considering that that number is shared between the number of sockets, no?

When a server connects as a client then there is not that many sockets... This is relevant for client side.

[ronag]

140 ms

I have randomFillSync calls that take up to 300 ms 😢

[sdrsdr]

I can imagine a server system initiating a websocket conections to other servers for example forming a mesh but how big a mesh can be? Not really a problem in need of solving ATM

[ronag]

@lpinca Does the mask need to be cryptographically random or would pseudo random numbers also be ok here?

[lpinca]

cryptographically random

This.

Clients MUST choose a new masking key for each frame, using an
algorithm that cannot be predicted by end applications that provide
data. For example, each masking could be drawn from a
cryptographically strong random number generator.

[sdrsdr]

https://stackoverflow.com/questions/14174184/what-is-the-mask-in-a-websocket-frame

Why is there masking at all? Because apparently there is enough broken infrastructure out there
that lets the upgrade header go through and then handles the rest of the connection as a second 
HTTP request which it then stuffs into the cache. I have no words for this. In any case, the defence 
against that is basically a strong 32bit random number as masking key.```

[lpinca]

I have randomFillSync calls that take up to 300 ms.

That's really weird. Is it the same with crypto.randomBytes() sync? I guess the underlying code is the same.

[ronag]

Is it the same with crypto.randomBytes() sync? I guess the underlying code is the same.

Yea, it's the same.

[ronag]

I solved this in production buy doing send(data, { mask: false })

[ronag]

I solved this in production buy doing send(data, { mask: false })

Actually... that doesn't work at all.

[lpinca]

Yes the server must close the connection if an unmasked frame is received.

[sdrsdr]

Haha! I see a spot here for "ignore-the-standards-and-be-fast" setup if both client and server are willing to disobey the rules

[ronag]

Another workaround. Move ws to a separate thread.

// socket-worker.js
'use strict'

const { Worker } = require('worker_threads')
const path = require('path')

class Socket {
  constructor(url) {
    this._worker = new Worker(path.join(__dirname, 'socket-worker.js'), {
      workerData: {
        url: url.href,
      },
    })
      .on('error', (err) => {
        this.onerror?.(err)
      })
      .on('exit', () => {
        this.readyState = this.CLOSED
      })
      .on('message', ({ event, data }) => {
        if (event === 'open') {
          this.readyState = this.OPEN
          this.onopen?.()
        } else if (event === 'error') {
          this.readyState = this.CLOSING
          this.onerror?.(data)
        } else if (event === 'close') {
          this.readyState = this.CLOSED
          this.onclose?.()
        } else if (event === 'data') {
          this.onmessage?.({ data: Buffer.from(data) })
        }
      })

    this.CONNECTING = 'CONNECTING'
    this.OPEN = 'OPEN'
    this.CLOSING = 'CLOSING'
    this.CLOSED = 'CLOSED'

    this.onopen = null
    this.onerror = null
    this.onclose = null
    this.onmessage = null
    this.readyState = this.CONNECTING
  }

  send(data) {
    // TODO (perf): Transfer Buffer?
    this._worker.postMessage(data)
  }

  close() {
    this.readyState = this.CLOSING
    this._worker.terminate()
  }
}

module.exports = Socket

// socket.js
'use strict'

const { Worker } = require('worker_threads')
const path = require('path')

class Socket {
  constructor(url) {
    this._worker = new Worker(path.join(__dirname, 'socket-worker.js'), {
      workerData: {
        url: url.href,
      },
    })
      .on('error', (err) => {
        this.onerror?.(err)
      })
      .on('message', ({ event, data }) => {
        if (event === 'open') {
          this.readyState = this.OPEN
          this.onopen?.()
        } else if (event === 'error') {
          this.readyState = null
          this.onerror?.(data)
        } else if (event === 'close') {
          this.readyState = null
          this.onclose?.()
        } else if (event === 'data') {
          this.onmessage?.({ data: Buffer.from(data) })
        }
      })

    this.OPEN = 'OPEN'

    this.onopen = null
    this.onerror = null
    this.onclose = null
    this.onmessage = null
    this.readyState = null
  }

  send(data) {
    // TODO (perf): Transfer Buffer?
    this._worker.postMessage(data)
  }

  close() {
    this._worker.terminate()
  }
}

module.exports = Socket

[ronag]

We're good with disabling masking and/or worker thread. Solves our perf issues.

[lpinca]

@ronag are you running it on a resource constrained system? This is what I get on my machine:

const crypto = require('crypto');

const buffer = Buffer.alloc(4);

let max = 0n;

for (let i = 0; i < 1e6; i++) {
  const start = process.hrtime.bigint();
  crypto.randomFillSync(buffer, 0, 4);
  const end = process.hrtime.bigint();

  const elapsed = end - start;

  if (elapsed > max) {
    max = elapsed;
  }
}

console.log(max);

$ node test.js 
2040128n

That's 2 ms in the worst case.

[ronag]

Not really. It’s an EPYC server. Node 16.13

[lpinca]

For correctness I have to take this #1986 (comment) back. Recent benchmarks show that masking ~10 KiB of data in plain JavaScript is now comparable to acquiring 4 bytes of random data.

[lpinca]

If there is evidence (I'm still not convinced) that this improves things in production, then I'm fine with reopening and merging.

[ronag]

I think it gets slow once the entropy pool is emptied.