transformers/tests/models/bert
Matt ee0d001de7
Add a TF in-graph tokenizer for BERT (#17701)
* Add a TF in-graph tokenizer for BERT

* Add from_pretrained

* Add proper truncation, option handling to match other tokenizers

* Add proper imports and guards

* Add test, fix all the bugs exposed by said test

* Fix truncation of paired texts in graph mode, more test updates

* Small fixes, add a (very careful) test for savedmodel

* Add tensorflow-text dependency, make fixup

* Update documentation

* Update documentation

* make fixup

* Slight changes to tests

* Add some docstring examples

* Update tests

* Update tests and add proper lowercasing/normalization

* make fixup

* Add docstring for padding!

* Mark slow tests

* make fixup

* Fall back to BertTokenizerFast if BertTokenizer is unavailable

* Fall back to BertTokenizerFast if BertTokenizer is unavailable

* make fixup

* Properly handle tensorflow-text dummies
2022-06-27 12:06:21 +01:00
..
__init__.py Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
test_modeling_bert.py Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
test_modeling_flax_bert.py Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
test_modeling_tf_bert.py Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
test_tokenization_bert_tf.py Add a TF in-graph tokenizer for BERT (#17701) 2022-06-27 12:06:21 +01:00
test_tokenization_bert.py Black preview (#17217) 2022-05-12 16:25:55 -04:00