[BUG] pipeline engine is_gradient_accumulation_boundary is not set correctly

[BacharL]

Pipeline engine overrides deepspeed.runtime.engine.is_gradient_accumulation_boundary()
https://github.com/microsoft/DeepSpeed/blob/7e8bcc07d6532a380a7a439741609b058f083ed6/deepspeed/runtime/pipe/engine.py#L560
In this implementation is_gradient_accumulation_boundary will return True only when the pipe is executing _exec_optimizer_step or _exec_reduce_grads. It will return False during forward and backward calls. Additionally it will ignore calls to set_gradient_accumulation_boundary.

This behavior leads to multiple issue in deepspeed.runtine.engine.backward

1. loss list grows indefinitely
   https://github.com/microsoft/DeepSpeed/blob/7e8bcc07d6532a380a7a439741609b058f083ed6/deepspeed/runtime/engine.py#L1884

2. If zero is enabled optimizer is called with incorrect state of is_gradient_accumulation_boundary
   https://github.com/microsoft/DeepSpeed/blob/7e8bcc07d6532a380a7a439741609b058f083ed6/deepspeed/runtime/engine.py#L1894

3. amp loss scaling incorrect.
   https://github.com/microsoft/DeepSpeed/blob/7e8bcc07d6532a380a7a439741609b058f083ed6/deepspeed/runtime/engine.py#L1899C13-L1899C27

[jomayeri]

@hablb The first point has been fixed in a previous PR. Could you point to examples of the second too points causing issues?

[BacharL]

for #2 "Train/Samples/train_loss" will not be printed to tensorboard
https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/engine.py#L1910

for #3. I'm not aware of any specific issue, but according to the comment in line 1923 it may be required
https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/runtime/engine.py#L1923

[jomayeri]

Thanks for pointing to these. We'll find a solution and link the PR here.

[jomayeri]

Are you not seeing the summary events? There is a separate engine in runtime/pipe/engine.py that is used for Pipeline Parallelism. It overrides the backward function in runtime/engine.py and has an implementation that should be showing the summary events