riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-04 13:20:12 +06:00
Code Issues Actions 21 Packages Projects Releases Wiki Activity
8cfcfe58c0
transformers/docs/source/zh/internal/tokenization_utils.md
jiaqiw09 a502b0d427
translate internal folder files to chinese (#27638) 
* translate

* update

* update

---------

Co-authored-by: jiaqiw <wangjiaqi50@huawei.com>
2023-12-04 10:04:28 -08:00

1.5 KiB
Raw Blame History

Tokenizers的工具

并保留格式：此页面列出了tokenizers使用的所有实用函数，主要是类 [~tokenization_utils_base.PreTrained TokenizerBase] 实现了常用方法之间的 [PreTrained Tokenizer] 和 [PreTrained TokenizerFast] 以及混合类 [~tokenization_utils_base.SpecialTokens Mixin]。

其中大多数只有在您研究库中tokenizers的代码时才有用。

PreTrainedTokenizerBase

autodoc tokenization_utils_base.PreTrainedTokenizerBase - call - all

SpecialTokensMixin

autodoc tokenization_utils_base.SpecialTokensMixin

Enums和namedtuples(命名元组)

autodoc tokenization_utils_base.TruncationStrategy

autodoc tokenization_utils_base.CharSpan

autodoc tokenization_utils_base.TokenSpan