Thomas Wolf
827d6d6ef0
Cleanup fast tokenizers integration ( #3706 )
...
* First pass on utility classes and python tokenizers
* finishing cleanup pass
* style and quality
* Fix tests
* Updating following @mfuntowicz comment
* style and quality
* Fix Roberta
* fix batch_size/seq_length inBatchEncoding
* add alignement methods + tests
* Fix OpenAI and Transfo-XL tokenizers
* adding trim_offsets=True default for GPT2 et RoBERTa
* style and quality
* fix tests
* add_prefix_space in roberta
* bump up tokenizers to rc7
* style
* unfortunately tensorfow does like these - removing shape/seq_len for now
* Update src/transformers/tokenization_utils.py
Co-Authored-By: Stefan Schweter <stefan@schweter.it>
* Adding doc and docstrings
* making flake8 happy
Co-authored-by: Stefan Schweter <stefan@schweter.it>
2020-04-18 13:43:57 +02:00
Patrick von Platen
d22894dfd4
[Docs] Add DialoGPT ( #3755 )
...
* add dialoGPT
* update README.md
* fix conflict
* update readme
* add code links to docs
* Update README.md
* Update dialo_gpt2.rst
* Update pretrained_models.rst
* Update docs/source/model_doc/dialo_gpt2.rst
Co-Authored-By: Julien Chaumond <chaumond@gmail.com>
* change filename of dialogpt
Co-authored-by: Julien Chaumond <chaumond@gmail.com>
2020-04-16 09:04:32 +02:00
Lysandre Debut
bb7c468520
Documentation ( #2989 )
...
* All Tokenizers
BertTokenizer + few fixes
RobertaTokenizer
OpenAIGPTTokenizer + Fixes
GPT2Tokenizer + fixes
TransfoXLTokenizer
Correct rst for TransformerXL
XLMTokenizer + fixes
XLNet Tokenizer + Style
DistilBERT + Fix XLNet RST
CTRLTokenizer
CamemBERT Tokenizer
FlaubertTokenizer
XLMRobertaTokenizer
cleanup
* cleanup
2020-02-25 18:43:36 -05:00
Lysandre
24d5ad1dcc
Run the examples in slow
2020-01-23 09:38:45 -05:00
Lysandre
9ddf60b694
Tips + whitespaces
2020-01-23 09:38:45 -05:00
Lysandre
850795c487
Pytorch GPT
2020-01-23 09:38:45 -05:00
alberduris
81d6841b4b
GPU text generation: mMoved the encoded_prompt to correct device
2020-01-06 15:11:12 +01:00
alberduris
dd4df80f0b
Moved the encoded_prompts to correct device
2020-01-06 15:11:12 +01:00
LysandreJik
927904bc91
[doc] pytorch_transformers -> transformers
2019-09-26 08:47:15 -04:00
LysandreJik
4acd87ff4e
TF models added to documentation
2019-09-26 07:45:40 -04:00
thomwolf
31c23bd5ee
[BIG] pytorch-transformers => transformers
2019-09-26 10:15:53 +02:00
thomwolf
2397f958f9
updating examples and doc
2019-07-14 23:20:10 +02:00
LysandreJik
f773faa258
Fixed all links. Removed TPU. Changed CLI to Converting TF models. Many minor formatting adjustments. Added "TODO Lysandre filled" where necessary.
2019-07-10 14:45:56 -04:00
LysandreJik
8fe2c9d98e
Refactored Docstrings of BERT, GPT2, GPT, TransfoXL, XLM and XLNet.
2019-07-09 15:55:31 -04:00
LysandreJik
64fd986376
Tokenizers and Config classes are referenced.
2019-07-05 17:44:59 -04:00
LysandreJik
df759114c9
Single file documentation for each model, accompanied by the Documentation overview.
2019-07-05 17:35:26 -04:00
