How to increase the context length? #70

Activity

1. lostella

   added

   FAQFrequently asked questionFrequently asked question

   on May 12, 2024
2. lostella commented on May 12, 2024

   

   lostella

   on May 12, 2024

   Contributor

   More actions

   Hi @Riofd. Indeed, the model internally limits the context length (currently happens here).

   You can increase the model context length after loading it, as in the example below. However the models were trained with windows of data of limited length, so they may not be able to make sense of the increased context: it's something for experiments to verify. I believe that for proper handling of longer context, we will need to pretrain the models with longer windows of data, but let us know if the following works!

   Here I use pipeline.embed to be able to inspect the encoder output shape, but this would have the same effect if you do pipeline.predict instead:

   import torch
   from chronos import ChronosPipeline
   
   pipeline = ChronosPipeline.from_pretrained("amazon/chronos-t5-tiny")
   context = torch.ones(size=(2000,))
   
   # get encoder embeddings
   embedding, _ = pipeline.embed(context=context)
   print(embedding.shape)
   
   # patch the context length
   pipeline.tokenizer.config.context_length = 1024
   
   # get encoder embeddings again
   embedding, _ = pipeline.embed(context=context)
   print(embedding.shape)

   outputs

   torch.Size([1, 513, 256])
   torch.Size([1, 1025, 256])
   

   where 513 is 512 (the original model context length) + the EOS token embedding, and similarly 1025 after the patching.
3. Riofd commented on May 13, 2024

   

   Riofd

   on May 13, 2024

   Author

   More actions

   Thank you for your reply. I try your way and successfully get different result comparing with context_length=512, but the forecasting performance decreases significantly. Maybe I should try to fine-tune it?
4. lostella commented on May 13, 2024

   

   lostella

   on May 13, 2024

   Contributor

   More actions

   Yes, fine-tuning may be the best way to go. We added code and some instructions here, let me know if you run into any issue
5. Riofd

   closed this as completedon May 30, 2024
6. abdulfatir commented on May 30, 2024

   

   abdulfatir

   on May 30, 2024

   Contributor

   More actions

   Let's keep this open as a FAQ.
7. abdulfatir

   reopened this on May 30, 2024