Skip to content

[Bug]: Python API Run.logs() returns nothing for a finished run #4175

Description

@r4victor

Steps to reproduce

from dstack.api import Client
c = Client.from_config()
r = c.runs.get("<a run that has finished>")
print(len(b"".join(r.logs())))                 # 0
print(len(b"".join(r.logs(replica_num=0))))    # the actual logs

Actual behaviour

logs() yields nothing for any terminal run - no exception, no warning. dstack logs <same run> prints the logs in full.

_find_job matches on replica_num alone when it is given, but when it is None it matches only a submission whose status is RUNNING:

def _find_job(self, replica_num: Optional[int], job_num: int) -> Optional[Job]:
for j in self._run.jobs:
if (
replica_num is not None
and j.job_spec.replica_num == replica_num
or replica_num is None
and j.job_submissions[-1].status == JobStatus.RUNNING
) and j.job_spec.job_num == job_num:
return j
return None

Once a run is terminal nothing is RUNNING, so _find_job returns None and logs() returns an empty generator. The CLI is unaffected because its --replica argparse default is 0, so it never passes None.

Expected behaviour

logs() with default arguments should return the logs of a finished run, as the CLI does - fall back to replica 0 when no submission is running, or raise instead of yielding nothing.

dstack version

master at 2517069, server run from source.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions