Repository navigation

Work on the BPE tokenizer - #3252

Merged

goerch merged 35 commits into

ggml-org:masterfrom

goerch:falcon-tokenizer

Oct 3, 2023

Merged

Work on the BPE tokenizer#3252
goerch merged 35 commits into
ggml-org:masterfrom
goerch:falcon-tokenizer

Commits on Sep 18, 2023
- Work on the BPE tokenizer
  goerch
  committed
  bfaab6f
- Try to fix build problem
  goerch
  committed
  89e74c6
- Fix debug assertion failure
  goerch
  committed
  7770423
- Fix MSVC Unicode BOM problem
  goerch
  committed
  37cf135
- Cleanup and an improvement
  goerch
  committed
  91a527a
- Fix compiler warning
  goerch
  committed
  208d3d7
- Cleanup
  goerch
  committed
  407f76d
- Test doesn't work over the full range of Unicodes
  goerch
  committed
  311fcf1
- Update .gitignore and Makefile
  goerch
  committed
  c85cb29
- Another Makefile rule
  goerch
  committed
  048e659
Commits on Sep 19, 2023
- Testing Aquila
  goerch
  committed
  c0990bb
- Moving byte decoding back to `token_to_piece` ...
  goerch
  committed
  1b7c369
- Guarding some unusable code pathes
  goerch
  committed
  a4e9448
- Streamlining code and adding some more assertions
  goerch
  committed
  17ca832
- Adding a comment
  goerch
  committed
  4abbfb5
- Adding another assertion
  goerch
  committed
  59a30b7
- Fixed vocabulary guarding assertions
  goerch
  committed
  a6070b7
Commits on Sep 29, 2023
- Merge branch 'master' into falcon-tokenizer
  goerch
  authored
  16c06fe
- Fix PR for recent change
  goerch
  authored
  c09330e
- Fix PR for recent change
  goerch
  authored
  9cfb714
- Fix for compiler warning
  goerch
  authored
  607e3bf
- Fix PR for recent change
  goerch
  authored
  fad8a77
- Fix PR for recent change
  goerch
  authored
  6a16c36
- Fix PR for recent change
  goerch
  authored
  a2ddaad
- Fix for compiler warning
  goerch
  authored
  3fa8c55
- Fixes for more compiler warnings
  goerch
  authored
  d6d7d0f
Commits on Oct 1, 2023
- Remove unused code
  goerch
  authored
  37af613
- Fix initialization of static maps
  goerch
  authored
  2117e23
Commits on Oct 2, 2023
- Add scores and token types back, adapt gptneox
  goerch
  authored
  28778f8
- Update llama.cpp
  
  goerch
  and
  ggerganov
  authored
  a9a2af9
- Update unicode.h
  
  goerch
  and
  ggerganov
  authored
  dccd1db
- Update unicode.h
  
  goerch
  and
  ggerganov
  authored
  02b9ccf
- Ported Starcoder and added some assertions
  goerch
  authored
  3d162cc
- Fix coding style
  goerch
  authored
  5aee498
Commits on Oct 3, 2023
- Apply @jploski 's fix for missing tokens
  goerch
  authored
  3e518e2

Commits

Commits on Sep 18, 2023

Commits on Sep 19, 2023

Commits on Sep 29, 2023

Commits on Oct 1, 2023

Commits on Oct 2, 2023

Commits on Oct 3, 2023