Skip to content

[MAIN-553] Create explicit executors to limit threadcount - #82

Merged
Avi-Robusta merged 5 commits into
mainfrom
limit-parallel-executors
Jun 27, 2023
Merged

Avi-Robusta merged 5 commits into
mainfrom
limit-parallel-executors

Conversation

@LeaveMyYard

Copy link
Copy Markdown
Contributor

Replace asyncio.to_thread to loop.run_in_executor to explicitly provide executor.
Default python max_thread_count turned out to eat away lots of resources or even kill prometheus on big k8s nodes.

Note:

  • Executor for runner running calculations when the data is collected is set to 6
  • Executor for Kubernetes API fetcher is set to 6, but there is one for each cluster
  • Executor for Prometheus API fetcher is set to 6, but there is one for each cluster

@LeaveMyYard LeaveMyYard self-assigned this Jun 22, 2023
Comment thread robusta_krr/core/runner.py Resolved
@Avi-Robusta Avi-Robusta reopened this Jun 25, 2023

@Avi-Robusta Avi-Robusta left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I had one thing I wanted clarity for.
Also I thought we discussed also adding rate limiting in the meeting too.
This could cause break on runner so I would like you to check the time it takes to run on average since by default runner waits only five minutes

@LeaveMyYard

Copy link
Copy Markdown
Contributor Author

Looks good, I had one thing I wanted clarity for. Also I thought we discussed also adding rate limiting in the meeting too. This could cause break on runner so I would like you to check the time it takes to run on average since by default runner waits only five minutes

I have tested the different multithreading settings and the result is:
Lower multithreading count results in less loading on prometheus memory usage, as well as slowing down the scan process (1 thread scanning 44 objects was working 1m11s).

For bigger count of threads the speed increased. 6 threads consumed more memory of prometheus then 32, but I think we need to have a bigger test with a bigger cluster (32+ cores for a node, 300+ pods).

I am providing a CLI argument to change the amount of max workers. If bigger tests show that 6 is still to much, we will be able to decrease the default value, but for now it should do the work.

@LeaveMyYard

Copy link
Copy Markdown
Contributor Author

Keep in mind that performance is soon to be optimized by decreasing the amount of queries to prometheus.

@LeaveMyYard
LeaveMyYard requested a review from Avi-Robusta June 26, 2023 21:08
@Avi-Robusta
Avi-Robusta merged commit 9d1617e into main Jun 27, 2023
@Avi-Robusta
Avi-Robusta deleted the limit-parallel-executors branch June 27, 2023 06:39
@lujiajing1126 lujiajing1126 mentioned this pull request Jul 3, 2023
1 of 3 tasks
Avi-Robusta added a commit that referenced this pull request Oct 13, 2026
## Summary
- **idna** 3.7 -> 3.15 in `pyproject.toml`, `poetry.lock` and
`requirements.txt`: CVE-2026-45409 (Medium). Closes dependabot alerts
#82, #83, #84.
- **Dockerfile**: remove ensurepip's bundled
`pip-25.0.1-py3-none-any.whl` from `python:3.12-slim`. The installed pip
is already upgraded to 26.x, but the leftover wheel is what Artifact
Registry flags on `devel/krr` for CVE-2025-8869, CVE-2026-3219,
CVE-2026-6357, CVE-2026-8643 (Medium).

## Verification
- `poetry lock --no-update` (Poetry 1.8.5): only idna changes in the
lock
- `pytest`: 55 passed
- Built the image (`linux/arm64`): idna 3.15, pip 26.2.1,
`ensurepip/_bundled/` empty, `import robusta_krr` and `krr.py --help`
work
- `grype` on the new image: no matches for idna, pip or any of the CVEs
above (the current `devel/krr:fix_cves_2` image shows
GHSA-65pc-fj4g-8rjx on idna 3.7)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: avi@robusta.dev <avi@robusta.dev>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants