* add philosophy doc
* fix typos
* update doc
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* address Patricks suggestions
* add a training example and fix typos
* jit the training step
* jit train step
* fix example code
* typo
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* First pass
* More progress
* Add support for local attention
* More improvements
* More improvements
* Conversion script working
* Add CanineTokenizer
* Make style & quality
* First draft of integration test
* Remove decoder test
* Improve tests
* Add documentation
* Mostly docs improvements
* Add CanineTokenizer tests
* Fix most tests on GPU, improve upsampling projection
* Address most comments by @dhgarrette
* Remove decoder logic
* Improve Canine tests, improve docs of CanineConfig
* All tokenizer tests passing
* Make fix-copies and fix tokenizer tests
* Fix test_model_outputs_equivalence test
* Apply suggestions from @sgugger's review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Address some more comments
* Add support for hidden_states and attentions of shallow encoders
* Define custom CanineModelOutputWithPooling, tests pass
* First pass
* More progress
* Add support for local attention
* More improvements
* More improvements
* Conversion script working
* Add CanineTokenizer
* Make style & quality
* First draft of integration test
* Remove decoder test
* Improve tests
* Add documentation
* Mostly docs improvements
* Add CanineTokenizer tests
* Fix most tests on GPU, improve upsampling projection
* Address most comments by @dhgarrette
* Remove decoder logic
* Improve Canine tests, improve docs of CanineConfig
* All tokenizer tests passing
* Make fix-copies and fix tokenizer tests
* Fix test_model_outputs_equivalence test
* Apply suggestions from @sgugger's review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Address some more comments
* Make conversion script work for Canine-c too
* Fix tokenizer tests
* Remove file
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Add option to save on each training node
* Apply suggestions from code review
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Address review comments
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* [WIP] Easily train a new fast tokenizer from a given one
* Fix test
* Roll out to other tokenizers and add tests
* Fix bug with unk id and add emoji to test
* Really use something different in test
* Implement special tokens map
* Map special tokens in the Transformers tokenizers
* Fix test
* Make test more robust
* Fix test for BPE
* More robust map and test
Co-authored-by SaulLu
* Test file
* Stronger tests
Co-authored-by: SaulLu <lucilesaul.com@gmail.com>
* Map unk token for Wordpiece and address review comment
* Fix lowercase test and address review comment
* Fix all tests
* Simplify test
* Fix tests for realsies
* Easily train a new fast tokenizer from a given one - tackle the special tokens format (str or AddedToken) (#12420)
* Propose change in tests regarding lower case
* add new test for special tokens types
* put back the test part about decoding
* add feature: the AddedToken is re-build with the different mapped content
* Address review comment: simplify AddedToken building
Co-authored-by: sgugger <sylvain.gugger@gmail.com>
* Update src/transformers/tokenization_utils_fast.py
Co-authored-by: sgugger <sylvain.gugger@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: SaulLu <lucilesaul.com@gmail.com>
Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>
* Tensorflow MLM example
* Add CLM example
* Style fixes, adding missing checkpoint code from the CLM example
* Fix TPU training, avoid massive dataset warnings
* Fix incorrect training length calculation for multi-GPU training
* Fix incorrect training length calculation for multi-GPU training
* Refactors and nitpicks from the review
* Style pass
* Adding README
Before the code could not be used for validation only because of this line:
extension = data_args.train_file.split(".")[-1]
was assuming that extension must be extracted from the training dataset. This line would run regardless of the training or validation options of the user. This would lead to an error if the user only wants to run an evaluation only and does not want to do train (because the training file does not exist). I modified it to extract extension from the training file if the user wants to do train and extract it from the validation file if the user wants to run eval. This way the code can be used for both training and validation separately.
* fixed multiplechoice tokenization
The model would have seen two sequences:
1. [CLS]prompt[SEP]prompt[SEP]
2. [CLS]choice0[SEP]choice1[SEP]
that is not correct as we want a contextualized embedding of prompt and choice
* removed outer brackets for proper sequence generation
