Repository navigation

convert : add t5 tokenizer tests - #1

Merged

fairydreaming merged 3 commits into

fairydreaming:t5-clean-3from

ggml-org:fairydreaming/t5-clean-3-gg

Jul 2, 2024

Merged

convert : add t5 tokenizer tests#1
fairydreaming merged 3 commits into
fairydreaming:t5-clean-3from
ggml-org:fairydreaming/t5-clean-3-gg

Commits on Jul 2, 2024
- convert : add t5 tokenizer tests
  ggerganov
  committed
  9eb5d56
- llama : UGM tokenizer init with UNK tokens instead of PAD
  ggerganov
  committed
  17bb0ea
- convert : use non-fast T5 tokenizer
  ggerganov
  committed
  703764a