Skip to content

[Tokenizers] Port BERTTokenizers #6991

Closed
Closed
[Tokenizers] Port BERTTokenizers#6991
Labels
enhancementNew feature or request
Milestone
ML.NET 5.0

Description

@ericstj
ericstj
opened on Feb 8, 2024

Porting BERTTokenizers enables several text embedding generation models. Requires #6988.

https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#text-embeddings.
https://github.com/huggingface/transformers/blob/v4.37.0/src/transformers/models/bert/tokenization_bert.py#L137

cc @luisquintanilla

We already have some BERT implementation which may be sufficient.

Activity

  1. ericstj
    added
    enhancementNew feature or request
    P2
    on Feb 8, 2024
  2. ericstj
    added this to the ML.NET Future milestone on Feb 8, 2024
  3. ericstj
    modified the milestones: ML.NET Future, ML.NET 4.0 on Aug 19, 2024
  4. ericstj
    added
    P1
    and removed
    P2
    on Aug 19, 2024

  5. ds5678 commented on Nov 13, 2024

    @ds5678
    ds5678
    on Nov 13, 2024

    Has this been implemented? It seems very similar to #7281.

  6. ericstj
    modified the milestones: ML.NET 4.0, ML.NET 5.0 on Jan 16, 2025
  7. tarekgh
    closed this as completedon Oct 29, 2025
  8. github-actions
    locked and limited conversation to collaborators on Nov 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions