[shim] Don't check image downloaded size - #2903
Merged
un-def merged 1 commit intoJul 15, 2025
Merged
Conversation
Sometimes dockerd emits less "Download complete" messages than
expected, but the image is pulled successfully.
The reason is unclear, but, anyway, this check is redundant since
we also rely on the status message, which is emitted only
in the case of succesfull pull.
In addition, this patch adds/changes the following:
* Write dockerd pull stream (JSON Lines) to {runnerDir}/pull.log —
useful for debugging (in conjunction with
DSTACK_SERVER_KEEP_SHIM_TASKS=1)
* Use `errorDetail.message` instead of deprecated `error`
* Move `ctx.Err()` check upper, otherwise it's shadowed by pull errors
Fixes: #2503
r4victor
approved these changes
Jul 15, 2025
un-def
added a commit
that referenced
this pull request
Aug 28, 2026
`~/.dstack/runners/<name>` is mounted into the task container as `/tmp/runner`. It used to hold runner's files only, but shim now keeps its own files there as well -- the image pull log since #2903 and the task state file since #4220 -- sharing them with the runner and the user workload, which may corrupt or delete them. Nothing sensitive is stored there today, but the approach is unsafe: nothing stops a contributor from putting a secret into a file the container can read. That dir is now the task dir, private to shim, and only its new `runner` subdir is mounted into the container: ~/.dstack/runners/<container-name>/ 0700, shim only task.json pull.log runner/ 0755, mounted as /tmp/runner * `runnerDir`/`runnersDir` are renamed to `taskDir`/`tasksDir` throughout, including the `DockerParameters` methods, to signal that the dir is managed by shim rather than by runner. The `runners` path itself is kept: an upgraded shim must find the dirs of the tasks created by the previous version. * The dir of a restored task is no longer derived from the container mounts, which now point at the `runner` subdir, but from the task ID in its state file. The tasks dir is scanned once on start, and the result is shared by the restore and the orphan sweep, which used to scan the dir a second time. * A task started by a shim version that did not write state files cannot be found by its state file, so its dir, named after the container, is looked up by name and, as before, removed along with the task. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sometimes dockerd emits less "Download complete" messages than expected, but the image is pulled successfully.
The reason is unclear, but, anyway, this check is redundant since we also rely on the status message, which is emitted only in the case of succesfull pull.
In addition, this patch adds/changes the following:
errorDetail.messageinstead of deprecatederrorctx.Err()check upper, otherwise it's shadowed by pull errorsFixes: #2503