Repository navigation

feat: add tokenizer module with sync and async support - #2012

Merged

dirkkul merged 9 commits into

feat/tokenizer-endpoint

Apr 16, 2026

Merged

feat: add tokenizer module with sync and async support#2012
dirkkul merged 9 commits into
dev/1.37from
feat/tokenizer-endpoint

Commits on Apr 14, 2026
- feat: add tokenizer module with sync and async support, including integration tests
  amourao
  committed
  239ed32
Commits on Apr 15, 2026
- refactor: names don't shadow existing
  amourao
  committed
  8b2caaf
- fix: add version gate
  amourao
  committed
  ede0b96
- refactor: update tokenization type to use Tokenization enum in TokenizeResult and related tests
  amourao
  committed
  8d379f4
- refactor: models
  amourao
  committed
  91a359a
- refactor: move tokenize property to class config
  amourao
  committed
  61665e7
- Merge branch 'dev/1.37' into feat/tokenizer-endpoint
  amourao
  authored
  3f78571
- fix: remove trailing whitespace in __init__.py
  amourao
  committed
  aea0327
- test: add version gate for Weaviate >= 1.37.0 in tokenization tests
  amourao
  committed
  ef55ce2