train_batch_size + dataset + actual batch size

[agemagician]

Hello,

I have 4 questions for clarification:

1. Why we should pass the training_data to the deepspeed.initialize to generate a new trainloader rather than using a normal torch trainloader ?
2. Can we use a custom pytorch trainloader in case we have custom dataset that returns for example inputs, outputs and mask ?
3. If the actual batch size that is used to be passed to the model is different than the train_batch_size in the json file, what will happen ?
4. Can we just define gradient_accumulation_steps and train_micro_batch_size_per_gpu
   only and leave deepspeed to calculate train_batch_size automatically ?

[tjruwase]

1. Passing training_data to deepspeed.initialize is optional, and not required. Some models can benefit from deepspeed's i/o optimizations. However, using torch trainloader is fine.

2. In our experience, deepspeed works well with custom trainloaders and datasets. So I suspect the answer to that is yes, but please share any issues you run into.

3. The train_batch_size in json file is used to perform gradient accumulation and compute progress statistics, so a mismatch could result in incorrect training and confusing statistics.

4. Yes, this is supported.

[msdejong]

@tjruwase could you clarify - how does the parallelization work with a custom dataloader? Do you need to make sure the dataloader uses the local rank as input to load a separate portion of the dataset manually?

[tjruwase]

This is correct, it is the responsibility of the custom loader to load the appropriate dataset portion based on local rank (or whatever else the client finds appropriate) into GPU memory. DeepSpeed does not impose any restrictions on the custom dataloader, and does not perform any sanity checks.

Hope that clears things up.

[agemagician]

@tjruwase Thanks for the clarification.
Just one more question.

Assuming I have a custom data generator like that:

for batch_idx, batch in enumerate(DatasetGenerator):
            data, target,src_padding = batch['input'].to(model_engine.local_rank), batch['target'].to(model_engine.local_rank), batch['padding_mask'].to(model_engine.local_rank)

Does this mean:

1. The batch size should be equal to train_micro_batch_size_per_gpu ?
2. It should provide different/random batch for each gpu/node ?

[tjruwase]

1. Yes, batch_size should be equal to train_micro_batch_size_per_gpu, which is batch size for a single step on one gpu.

2. Assuming DatasetGenerator is returning the correct batch for each gpu, then this would be correct since the .to() is simply moving the data bits into gpu memory.

[agemagician]

@tjruwase Thanks a lot that answers all my questions for now :)

[tnq177]

@tjruwase assuming I'm using some customer data loader as @agemagician above but in a multi-node, multi-gpu settings, how would I go about and send tensors to the right GPU? Do I still do tensor.to(engine.local_rank) or global_rank please? Thanks.

[tjruwase]

@tnq177, I believe you want local_rank as crossing node boundaries requires communication collectives, like broadcast.