Run ESLint in parallel for multiple Nx projects

[matejchalk]

User story

For large Nx monorepos, it could help speed up Code PushUp runs if ESLint could run on different projects in parallel instead of sequentially. Adding an option to enable parallel execution, and even set the max number of processes, would give Code PushUp users more flexibility to optimize for performance. A lot like Nx's --parallel flag.

Acceptance criteria

• new parallel?: boolean | number option is added to eslintPlugin initializer
• default is to lint projects sequentially - so effective default is parallel: false
• if an integer is specified (e.g. parallel: 4), then projects are chunked by this max value
  • within each chunk, processes are executed in parallel using Promise.all and executeProcess
  • chunks are executed sequentially
• parallel: true enables parallel execution with maximum set to os.cpus().length
• for future reusability, a generic function which orchestrates this sequential/parallel process execution is exported from @code-pushup/utils
• test performance impact on some large Nx monorepos
• document usage in @code-pushup/eslint-plugin

Implementation details

No response

[getlarge]

Have you considered using the worker_threads module from node to run in parallel? I would not be surprised if Nx would use a similar approach where the integer passed to parallel is the number of workers.

[getlarge]

Have you considered using the worker_threads module from node to run in parallel? I would not be surprised if Nx would use a similar approach where the integer passed to parallel is the number of workers.

My assumption was incorrect!

[matejchalk]

I guess that could also work. I'm honestly not sure what the pros/cons would be of multi-threading instead of multi-processing. Since we're already using processes on the project (eslint is executed as a process, and we have a convenient executeProcess wrapper which promisifes spawn from child_process module), they should be more straightforward to implement. But could be worth investigating threads as an alternative.

[vmasek]

There is a benefit to use multi threading as it is generally faster to spin up, has less overhead (memory & CPU resources) but it's biggest benefit shared memory access is also a risk.

I agree that spawning process is already established as go to approach in the codebase and it has some benefits, namely we do not risk memory race conditions or need to be afraid that error would kill the whole execution of the CLI.

I plan to give this issue a try in following weeks with a POC

[BioPhoton]

@matejchalk
This is most probably solved as we can now use artifacts options in the eslint plugin and offload all work, parallelization and caching to Nx.

[matejchalk]

This is most probably solved as we can now use artifacts options in the eslint plugin and offload all work, parallelization and caching to Nx.

Yes, for Nx (and also Turborepo) this is solved with the custom command. This feature may still be useful for non-Nx monorepos, but that's more niche. So I keep this issue open, but deprioritize it, I don't think we'll need it for the foreseeable future.