Skip to content

关于tokenizer所需内存的提示 #66

Description

@Liton-8

建议修改一下关于内存占用的说明,这里以20260601的wiki文件为例,2.01G左右大小的文档,实际跑train_tokenizer.py需要占用的内存大致在60G以上,本机空载占用8G内存的情况下跑这个已经触发数次内存交换压缩。
(PS:实际内存不需要这么多应该也能跑得动,但是如果无任何交换的情况下占用应该是十分可观的……)
Image

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions