mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-23 06:20:22 +06:00
![]() * Add a TF in-graph tokenizer for BERT * Add from_pretrained * Add proper truncation, option handling to match other tokenizers * Add proper imports and guards * Add test, fix all the bugs exposed by said test * Fix truncation of paired texts in graph mode, more test updates * Small fixes, add a (very careful) test for savedmodel * Add tensorflow-text dependency, make fixup * Update documentation * Update documentation * make fixup * Slight changes to tests * Add some docstring examples * Update tests * Update tests and add proper lowercasing/normalization * make fixup * Add docstring for padding! * Mark slow tests * make fixup * Fall back to BertTokenizerFast if BertTokenizer is unavailable * Fall back to BertTokenizerFast if BertTokenizer is unavailable * make fixup * Properly handle tensorflow-text dummies |
||
---|---|---|
.. | ||
__init__.py | ||
test_modeling_bert.py | ||
test_modeling_flax_bert.py | ||
test_modeling_tf_bert.py | ||
test_tokenization_bert_tf.py | ||
test_tokenization_bert.py |