Steps to reproduce
-
dstack apply -f preset.dstack.yml --fleet <fleet> with:
type: preset
name: my-preset
base: Qwen/Qwen3.8-27B
trials: 4
# Agentic shape: long context, short turns, most of it cache-hittable, small-scale concurrency
min_context_length: 262144
max_ttft: 5000
concurrency: 4
input_tokens: 131072
output_tokens: 512
shared_prefix_tokens: 130048
env:
- HF_TOKEN
-
Compare base and model in ~/.dstack/presets/<id>/final_report.json.
Actual behaviour
base is not enforced. The agent may serve any repo it chooses, including a model from a different family, and the preset is still verified and reported as successful.
Verification checks only that the service advertises the requested name and that the report echoes the requested base:
|
if service.model is None or service.model.name != preset_configuration.model.api_model_name: |
|
raise CLIError("Claude final service model name does not match the requested model") |
|
if configuration.model.allows_variant_selection: |
|
if report.base != configuration.model.api_model_name: |
|
raise CLIError("Claude final report base does not match the requested model") |
|
elif report.model != configuration.model.exact_repo: |
|
raise CLIError("Claude changed an exact model request") |
Observed with base: Qwen/Qwen3.8-27B: the report recorded model: Qwen/Qwen3.5-27B-GPTQ-Int4, a different model generation, and the exported service ran vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name Qwen/Qwen3.8-27B. Whether a substitution happens depends on the agent's judgment about what counts as a compatible repo, so it does not reproduce on every run.
Expected behaviour
base should constrain which repos are acceptable.
dstack version
master at 2f024c6 (source checkout; dstack --version reports 0.0.0). macOS 24.0.0.
Additional information
I suspect agent will use a different model if it fails to satisfy the requirements. In my case, it failed to meet TTFT on A100 for 2 trials and switched the model generation.
Steps to reproduce
dstack apply -f preset.dstack.yml --fleet <fleet>with:Compare
baseandmodelin~/.dstack/presets/<id>/final_report.json.Actual behaviour
baseis not enforced. The agent may serve any repo it chooses, including a model from a different family, and the preset is still verified and reported as successful.Verification checks only that the service advertises the requested name and that the report echoes the requested base:
dstack/src/dstack/_internal/cli/services/presets/verify.py
Lines 99 to 100 in 2f024c6
dstack/src/dstack/_internal/cli/services/presets/verify.py
Lines 142 to 146 in 2f024c6
Observed with
base: Qwen/Qwen3.8-27B: the report recordedmodel: Qwen/Qwen3.5-27B-GPTQ-Int4, a different model generation, and the exported service ranvllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name Qwen/Qwen3.8-27B. Whether a substitution happens depends on the agent's judgment about what counts as a compatible repo, so it does not reproduce on every run.Expected behaviour
baseshould constrain which repos are acceptable.dstack version
master at 2f024c6 (source checkout;
dstack --versionreports 0.0.0). macOS 24.0.0.Additional information
I suspect agent will use a different model if it fails to satisfy the requirements. In my case, it failed to meet TTFT on A100 for 2 trials and switched the model generation.