Tokenizer batch encode#17

thomasw21 wants to merge 2 commits into

bigscience-workshop:mainbigscience-workshop/Megatron-DeepSpeed:mainfrom

thomasw21:tokenizer_batch_encodethomasw21/Megatron-DeepSpeed:tokenizer_batch_encodeCopy head branch name to clipboard