Skip to content

Adding task pending time in MSQ reports - #15966

Merged
cryptoe merged 4 commits into
apache:masterfrom
cryptoe:task_pending_time_msq
Feb 27, 2024
Merged

cryptoe merged 4 commits into
apache:masterfrom
cryptoe:task_pending_time_msq

Conversation

@cryptoe

@cryptoe cryptoe commented Feb 26, 2024 •

Copy link
Copy Markdown
Contributor
  • Added a new field pendingMs in MSQ task reports. This helps in figuring out the exact run time of the MSQ worker tasks.
  • Fixed data races.

The payload now looks like

"workers": {
          "0": [
            {
              "workerId": "query-3dc0c45d-34d7-4b15-86c9-cdb2d3ebfc4e-worker0_0",
              "state": "SUCCESS",
              "durationMs": 15511,
              "pendingMs": 137
            }
          ]
        }

Release notes

A new field in MSQ task report which captures the milliseconds elapsed between when the worker task was first requested and when it fully started RUNNING. Actual work time can be calculated using actualWorkTimeMS = durationMs - pendingMs

@github-actions github-actions Bot added Area - Documentation Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Feb 26, 2024
}

@JsonProperty("pendingMs")
public long getPendingTimeInMS()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pendingTimeMillis would probably be a better name and more in line with rest of the names in Druid.

@cryptoe cryptoe Feb 26, 2024 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we already have durationMs as part of line:389 of this payload, having pendingTimeMillis would be weird. If you feel strongly about it, I can change.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using millis as convention in this class, so it's a bit unfortunate that we already have "durationMs" in the taskReport. Would the best thing be to call it pendingTimeMillis in the code and expose it in the report as getPendingTimeInMs. As a nit, I think it should be getPendingTimeInMs though.

@adarshsanjeev adarshsanjeev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, minor comments

final TaskTracker tracker = taskEntry.getValue();
if (!canceledWorkerTasks.contains(taskId)
&& (tracker.status == null || !tracker.status.getStatusCode().isComplete())) {
if ((!canceledWorkerTasks.contains(taskId))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this looks a bit weird due to the parenthesis and new lines.

Suggested change
if ((!canceledWorkerTasks.contains(taskId))
if (!canceledWorkerTasks.contains(taskId) && !tracker.isComplete()) {

@cryptoe cryptoe Feb 27, 2024 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally find 2 not statements a little tricky so hence like the parenthesis. Keeping it as is for now.

@cryptoe
cryptoe merged commit 5bb5b41 into apache:master Feb 27, 2024
@adarshsanjeev adarshsanjeev added this to the 30.0.0 milestone May 6, 2024
riovic918data pushed a commit to riovic918data/druid that referenced this pull request Jun 12, 2026
    Added a new field pendingMs in MSQ task reports. This helps in figuring out the exact run time of the MSQ worker tasks.
    Fixed data races.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Batch Ingestion Area - Documentation Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants