(Re)activate test-tokenizer-0-falcon #3233

Activity

1. ggerganov commented on Sep 17, 2023

   

   ggerganov

   on Sep 17, 2023

   Member

   More actions

   It's failing - no unicode support. So decided to disable it until we add it

   👍React with 👍1Reacted by goerch
2. goerch commented on Sep 17, 2023

   

   goerch

   on Sep 17, 2023

   Last edited by goerch

   ContributorAuthor

   More actions

   I'm trying to get something like test-tokenizer-1 for falcon working. Yeah, Unicode seems to be part of the problem.

   @ all concerned : if we want to do proper tokenization, please don't modify the original vocabulary and add

       GGML_ASSERT(vocab.id_to_token.size() == vocab.token_to_id.size());
   

   to your vocab-loading code, otherwise I don't see a chance.
3. goerch

   closed this as completedin #3252on Oct 3, 2023