'suspend' and 'resume' support for supervisors (kafka indexing service, materialized views) by clintropolis · Pull Request #6234 · apache/druid

clintropolis · 2018-08-26T07:53:54Z

changes:

extends SupervisorSpec interface to describe supervisors which support suspend/resume functionality controlled through the SupervisorManager, which will gracefully shutdown the supervisor and it's tasks, update it's SupervisorSpec with either a suspended or running state, and update with the toggled spec. Spec updates are provided by SupervisorSpec.createSuspendedSpec and SupervisorSpec.createRunningSpec respectively.
KafkaSupervisorSpec now supports suspend/resume functionality. The difference in behavior between 'running' and 'suspended' state is whether the supervisor will attempt to ensure that indexing tasks are or are not running respectively. Behavior is identical otherwise.
MaterializedViewSupervisorSpec now supports suspend/resume functionality. MaterializedViewSupervisor when suspended will effectively do nothing until resumed.
SupervisorResource now provides /druid/indexer/v1/supervisor/{id}/suspend and /druid/indexer/v1/supervisor/{id}/resume which are used to suspend/resume supervisors
Deprecated /druid/indexer/v1/supervisor/{id}/shutdown and moved it's functionality to /druid/indexer/v1/supervisor/{id}/terminate since 'shutdown' is ambiguous verbage for something that effectively stops a supervisor forever
Added ability to get all supervisor specs from /druid/indexer/v1/supervisor by supplying the 'full' query parameter /druid/indexer/v1/supervisor?full which will return a list of json objects of the form {"id":<id>, "spec":<SupervisorSpec>}
Updated overlord console ui to enable suspend/resume, and changed 'shutdown' to 'terminate'

clintropolis · 2018-08-27T20:57:28Z

Updated overlord ui to move supervisor status to own column after seeing it look terribad with multiple rows, now looks like this:

jon-wei · 2018-09-12T01:35:11Z


  void createNewTasks() throws JsonProcessingException
  {
+    if (spec.isSuspended()) {


Is this needed, since createNewTasks() is only called when not suspended?

I don't think so, I'll remove.

jon-wei · 2018-09-12T01:54:13Z

          '<a href="/druid/indexer/v1/supervisor/' + supervisorId + '/history">history</a>' +
+          (supervisorSpec.suspended ?
+           '<a style="padding-right:5px;" onclick="resumeSupervisor(\'' + supervisorId + '\');">resume</a>' :
+           '<a onclick="suspendSupervisor(\'' + supervisorId + '\');">suspend</a>'


this would show a suspend button for tasks that don't support suspend, though that's a moot point if all the supervisors support suspend

jon-wei

LGTM, I'll take another look after you update the PR to support suspend across all supervisors

changes: * introduces `SuspendableSupervisorSpec` interface to describe supervisors which support suspend/resume functionality controlled through the `SupervisorManager`, which will gracefully shutdown the supervisor and it's tasks, update it's `SupervisorSpec` with either a suspended or running state, and update with the toggled spec. Spec updates are provided by `SuspendableSupervisorSpec.createSuspendedSpec` and `SuspendableSupervisorSpec.createRunningSpec` respectively. * `KafkaSupervisorSpec` extends `SuspendableSupervisorSpec` and now supports suspend/resume functionality. The difference in behavior between 'running' and 'suspended' state is whether the supervisor will attempt to ensure that indexing tasks are or are not running respectively. Behavior is identical otherwise. * `SupervisorResource` now provides `/druid/indexer/v1/supervisor/{id}/suspend` and `/druid/indexer/v1/supervisor/{id}/resume` which are used to suspend/resume suspendable supervisors * Deprecated `/druid/indexer/v1/supervisor/{id}/shutdown` and moved it's functionality to `/druid/indexer/v1/supervisor/{id}/terminate` since 'shutdown' is ambiguous verbage for something that effectively stops a supervisor forever * Added ability to get all supervisor specs from `/druid/indexer/v1/supervisor` by supplying the 'full' query parameter `/druid/indexer/v1/supervisor?full` which will return a list of json objects of the form `{"id":<id>, "spec":<SupervisorSpec>}` * Updated overlord console ui to enable suspend/resume, and changed 'shutdown' to 'terminate'

…s not look like garbage

… to support suspend, more tests

leventov · 2018-10-18T15:41:29Z

+                authorizedSupervisorIds.stream()
+                                       .map(x -> ImmutableMap.<String, Object>builder()
+                                           .put("id", x)
+                                           .put("spec", manager.getSupervisorSpec(x).get())


Calls Optional.get() without a check

…e, materialized views) (apache#6234) * 'suspend' and 'resume' support for kafka indexing service changes: * introduces `SuspendableSupervisorSpec` interface to describe supervisors which support suspend/resume functionality controlled through the `SupervisorManager`, which will gracefully shutdown the supervisor and it's tasks, update it's `SupervisorSpec` with either a suspended or running state, and update with the toggled spec. Spec updates are provided by `SuspendableSupervisorSpec.createSuspendedSpec` and `SuspendableSupervisorSpec.createRunningSpec` respectively. * `KafkaSupervisorSpec` extends `SuspendableSupervisorSpec` and now supports suspend/resume functionality. The difference in behavior between 'running' and 'suspended' state is whether the supervisor will attempt to ensure that indexing tasks are or are not running respectively. Behavior is identical otherwise. * `SupervisorResource` now provides `/druid/indexer/v1/supervisor/{id}/suspend` and `/druid/indexer/v1/supervisor/{id}/resume` which are used to suspend/resume suspendable supervisors * Deprecated `/druid/indexer/v1/supervisor/{id}/shutdown` and moved it's functionality to `/druid/indexer/v1/supervisor/{id}/terminate` since 'shutdown' is ambiguous verbage for something that effectively stops a supervisor forever * Added ability to get all supervisor specs from `/druid/indexer/v1/supervisor` by supplying the 'full' query parameter `/druid/indexer/v1/supervisor?full` which will return a list of json objects of the form `{"id":<id>, "spec":<SupervisorSpec>}` * Updated overlord console ui to enable suspend/resume, and changed 'shutdown' to 'terminate' * move overlord console status to own column in supervisor table so does not look like garbage * spacing * padding * other kind of spacing * fix rebase fail * fix more better * all supervisors now suspendable, updated materialized view supervisor to support suspend, more tests * fix log

clintropolis force-pushed the kafka-supervisor-suspend-resume branch from ead8d13 to d9e8470 Compare August 30, 2018 20:18

gianm added this to the 0.13.0 milestone Aug 31, 2018

jihoonson added the Area - Streaming Ingestion label Sep 12, 2018

jon-wei reviewed Sep 12, 2018

View reviewed changes

clintropolis added 8 commits September 12, 2018 16:05

move overlord console status to own column in supervisor table so doe…

bc32a97

…s not look like garbage

spacing

46bd0b8

padding

7a05455

other kind of spacing

38cce48

fix rebase fail

096ad70

fix more better

f15ecd1

all supervisors now suspendable, updated materialized view supervisor…

f2f8ebd

… to support suspend, more tests

clintropolis force-pushed the kafka-supervisor-suspend-resume branch from 881aa08 to f2f8ebd Compare September 12, 2018 23:05

fix log

3ebe01b

clintropolis changed the title ~~'suspend' and 'resume' support for kafka indexing service~~ 'suspend' and 'resume' support for supervisors (kafka indexing service, materialized views) Sep 12, 2018

jon-wei approved these changes Sep 13, 2018

View reviewed changes

jon-wei merged commit 91a37c6 into apache:master Sep 13, 2018

clintropolis deleted the kafka-supervisor-suspend-resume branch September 13, 2018 21:57

clintropolis mentioned this pull request Sep 13, 2018

Add suspend|resume|terminate all supervisors endpoints. #6272

Merged

leventov reviewed Oct 18, 2018

View reviewed changes

dclim mentioned this pull request Nov 27, 2018

Druid 0.13.0-incubating release notes #6442

Closed

Navigation Menu

Repository navigation

'suspend' and 'resume' support for supervisors (kafka indexing service, materialized views) - #6234

'suspend' and 'resume' support for supervisors (kafka indexing service, materialized views)#6234
jon-wei merged 9 commits into
apache:masterfrom
clintropolis:kafka-supervisor-suspend-resume

clintropolis commented Aug 26, 2018 •

edited

Loading

Uh oh!

clintropolis commented Aug 27, 2018

Uh oh!

jon-wei Sep 12, 2018

Uh oh!

clintropolis Sep 12, 2018

Uh oh!

jon-wei Sep 12, 2018

Uh oh!

jon-wei left a comment

Uh oh!

leventov Oct 18, 2018

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

clintropolis commented Aug 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clintropolis commented Aug 27, 2018

Uh oh!

jon-wei Sep 12, 2018

Choose a reason for hiding this comment

Uh oh!

clintropolis Sep 12, 2018

Choose a reason for hiding this comment

Uh oh!

jon-wei Sep 12, 2018

Choose a reason for hiding this comment

Uh oh!

jon-wei left a comment

Choose a reason for hiding this comment

Uh oh!

leventov Oct 18, 2018

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

clintropolis commented Aug 26, 2018 •

edited

Loading