Skip to content

[Bug]: preset base is not enforced - the preset may use any model #4197

Description

@r4victor

Steps to reproduce

  1. dstack apply -f preset.dstack.yml --fleet <fleet> with:

    type: preset
    name: my-preset
    base: Qwen/Qwen3.8-27B
    trials: 4
    
    # Agentic shape: long context, short turns, most of it cache-hittable, small-scale concurrency
    min_context_length: 262144
    max_ttft: 5000
    concurrency: 4
    input_tokens: 131072
    output_tokens: 512
    shared_prefix_tokens: 130048
    
    env:
      - HF_TOKEN
    
  2. Compare base and model in ~/.dstack/presets/<id>/final_report.json.

Actual behaviour

base is not enforced. The agent may serve any repo it chooses, including a model from a different family, and the preset is still verified and reported as successful.

Verification checks only that the service advertises the requested name and that the report echoes the requested base:

if service.model is None or service.model.name != preset_configuration.model.api_model_name:
raise CLIError("Claude final service model name does not match the requested model")

if configuration.model.allows_variant_selection:
if report.base != configuration.model.api_model_name:
raise CLIError("Claude final report base does not match the requested model")
elif report.model != configuration.model.exact_repo:
raise CLIError("Claude changed an exact model request")

Observed with base: Qwen/Qwen3.8-27B: the report recorded model: Qwen/Qwen3.5-27B-GPTQ-Int4, a different model generation, and the exported service ran vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name Qwen/Qwen3.8-27B. Whether a substitution happens depends on the agent's judgment about what counts as a compatible repo, so it does not reproduce on every run.

Expected behaviour

base should constrain which repos are acceptable.

dstack version

master at 2f024c6 (source checkout; dstack --version reports 0.0.0). macOS 24.0.0.

Additional information

I suspect agent will use a different model if it fails to satisfy the requirements. In my case, it failed to meet TTFT on A100 for 2 trials and switched the model generation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions