3 Commits

Author	SHA1	Message	Date
Yih-Dar	b3a0aad37d	Fix past CI (#20967) ... * Fix for Past CI * make style * clean up * unindent 2 blocks Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-01-12 18:04:21 +01:00
Pi Esposito	0e0b7cb72a	Allow usage of TF Text BertTokenizer on TFBertTokenizer to make it servable on TF Serving (#19590) ... * add suport for non fast tf bert tokenizer * add tests for non fast tf bert tokenizer * fix fast bert tf tokenizer flag * double tokenizers list on tf tokenizers test to aovid breaking zip on test output equivalence * reformat code with black to comply with code quality checks * trigger ci	2022-10-14 15:18:02 +01:00
Matt	ee0d001de7	Add a TF in-graph tokenizer for BERT (#17701) ... * Add a TF in-graph tokenizer for BERT * Add from_pretrained * Add proper truncation, option handling to match other tokenizers * Add proper imports and guards * Add test, fix all the bugs exposed by said test * Fix truncation of paired texts in graph mode, more test updates * Small fixes, add a (very careful) test for savedmodel * Add tensorflow-text dependency, make fixup * Update documentation * Update documentation * make fixup * Slight changes to tests * Add some docstring examples * Update tests * Update tests and add proper lowercasing/normalization * make fixup * Add docstring for padding! * Mark slow tests * make fixup * Fall back to BertTokenizerFast if BertTokenizer is unavailable * Fall back to BertTokenizerFast if BertTokenizer is unavailable * make fixup * Properly handle tensorflow-text dummies	2022-06-27 12:06:21 +01:00