riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-23 06:20:22 +06:00
Code Issues Actions 34 Packages Projects Releases Wiki Activity
3,771 Commits 880 Branches 229 Tags 2 GiB
1dc9b3c784
Commit Graph

221 Commits

Author SHA1 Message Date
Julien Chaumond
1dc9b3c784 Fixes #3877 2020-04-22 01:15:10 +00:00
Thomas Wolf
827d6d6ef0
Cleanup fast tokenizers integration (#3706) 
* First pass on utility classes and python tokenizers

* finishing cleanup pass

* style and quality

* Fix tests

* Updating following @mfuntowicz comment

* style and quality

* Fix Roberta

* fix batch_size/seq_length inBatchEncoding

* add alignement methods + tests

* Fix OpenAI and Transfo-XL tokenizers

* adding trim_offsets=True default for GPT2 et RoBERTa

* style and quality

* fix tests

* add_prefix_space in roberta

* bump up tokenizers to rc7

* style

* unfortunately tensorfow does like these - removing shape/seq_len for now

* Update src/transformers/tokenization_utils.py

Co-Authored-By: Stefan Schweter <stefan@schweter.it>

* Adding doc and docstrings

* making flake8 happy

Co-authored-by: Stefan Schweter <stefan@schweter.it>
 2020-04-18 13:43:57 +02:00
Patrick von Platen
d22894dfd4
[Docs] Add DialoGPT (#3755) 
* add dialoGPT

* update README.md

* fix conflict

* update readme

* add code links to docs

* Update README.md

* Update dialo_gpt2.rst

* Update pretrained_models.rst

* Update docs/source/model_doc/dialo_gpt2.rst

Co-Authored-By: Julien Chaumond <chaumond@gmail.com>

* change filename of dialogpt

Co-authored-by: Julien Chaumond <chaumond@gmail.com>
 2020-04-16 09:04:32 +02:00
Julien Chaumond
cbad305ce6
[docs] The use of do_lower_case in scripts is on its way to deprecation (#3738) 2020-04-10 12:34:04 -04:00
Sam Shleifer
7a7fdf71f8
Multilingual BART - (#3602) 
- support mbart-en-ro weights
- add MBartTokenizer
 2020-04-10 11:25:39 -04:00
Lysandre Debut
261c4ff4e2
Update notebooks (#3620) 
* Update notebooks

* From local to global link

* from local links to *actual* global links
 2020-04-06 14:32:39 -04:00
LysandreJik
36bffc81b3 Release: v2.8.0 2020-04-06 10:03:53 -04:00
Julien Chaumond
94eb68d742 weigths*weights 2020-04-04 15:03:26 -04:00
Lysandre Debut
d5d7d88612
ELECTRA (#3257) 
* Electra wip

* helpers

* Electra wip

* Electra v1

* ELECTRA may be saved/loaded

* Generator & Discriminator

* Embedding size instead of halving the hidden size

* ELECTRA Tokenizer

* Revert BERT helpers

* ELECTRA Conversion script

* Archive maps

* PyTorch tests

* Start fixing tests

* Tests pass

* Same configuration for both models

* Compatible with base + large

* Simplification + weight tying

* Archives

* Auto + Renaming to standard names

* ELECTRA is uncased

* Tests

* Slight API changes

* Update tests

* wip

* ElectraForTokenClassification

* temp

* Simpler arch + tests

Removed ElectraForPreTraining which will be in a script

* Conversion script

* Auto model

* Update links to S3

* Split ElectraForPreTraining and ElectraForTokenClassification

* Actually test PreTraining model

* Remove num_labels from configuration

* wip

* wip

* From discriminator and generator to electra

* Slight API changes

* Better naming

* TensorFlow ELECTRA tests

* Accurate conversion script

* Added to conversion script

* Fast ELECTRA tokenizer

* Style

* Add ELECTRA to README

* Modeling Pytorch Doc + Real style

* TF Docs

* Docs

* Correct links

* Correct model intialized

* random fixes

* style

* Addressing Patrick's and Sam's comments

* Correct links in docs
 2020-04-03 14:10:54 -04:00
Patrick von Platen
83d1fbcff6
[Docs] Add usage examples for translation and summarization (#3538) 2020-03-31 09:36:03 -04:00
Patrick von Platen
42e1e3c67f
Update usage doc regarding generate fn (#3504) 2020-03-31 09:31:46 -04:00
LysandreJik
6f5a12a583 Release: v2.7.0 2020-03-30 08:49:24 -04:00
Patrick von Platen
5b44e0a31b
[T5] Add training documenation (#3507) 
* Add clear description of how to train T5

* correct docstring in T5

* correct typo

* correct docstring format

* update t5 model docs

* implement collins feedback

* fix typo and add more explanation for sentinal tokens

* delete unnecessary todos
 2020-03-30 13:35:53 +02:00
Patrick von Platen
fa9af2468a
Add T5 to docs (#3461) 
* add t5 docs basis

* improve docs

* add t5 docs

* improve t5 docstring

* add t5 tokenizer docstring

* finish docstring

* make style

* add pretrained models

* correct typo

* make examples work

* finalize docs
 2020-03-27 10:57:16 -04:00
LysandreJik
471cce24b3 Release: v2.6.0 2020-03-24 10:37:32 -04:00
Sam Shleifer
38a555a83c
Add Summarization to Pipelines (#3128) 
* passing

* Undo stupid chg

* docs

* undo rename

* delete-cruft

* only import if you have torch

* Dont rely on dict ordering

* Fix dict ordering upstream

* docstring link

* docstring link

* remove trailing comma for 3.5 compat

* new name

* delegate kwarging

* Update kwargs
 2020-03-17 18:04:21 -04:00
Thomas Wolf
2187c49f5c
CPU/GPU memory benchmarking utilities - Remove support for python 3.5 (now only 3.6+) (#3186) 
* memory benchmark rss

* have both forward pass and line-by-line mem tracing

* cleaned up tracing

* refactored and cleaning up API

* no f-strings yet...

* add GPU mem logging

* fix GPU memory monitoring

* style and quality

* clean up and doc

* update with comments

* Switching to python 3.6+

* fix quality
 2020-03-17 10:17:11 -04:00
Julien Chaumond
d6de6423ba [doc] --organization tweak 
Co-Authored-By: Thomas Wolf <thomwolf@users.noreply.github.com>
 2020-03-10 16:52:44 -04:00
Julien Chaumond
0e56dc3078 [doc] Document the new --organization flag of CLI 2020-03-10 16:42:01 -04:00
Sam Shleifer
857e0a0d3b
Rename BartForMaskedLM -> BartForConditionalGeneration (#3114) 
* improved documentation
 2020-03-05 17:41:18 -05:00
Lysandre
07a79db505 Fix failing doc samples 2020-03-04 19:11:31 -05:00
Lysandre Debut
d3eb7d23a4
Pipeline doc (#3055) 
* Pipeline doc initial commit

* pipeline abstraction

* Remove modelcard argument from pipeline

* Task-specific pipelines can be instantiated with no model or tokenizer

* All pipelines doc
 2020-03-02 14:07:10 -05:00
Sam Shleifer
b54ef78d0c
Bart-CNN (#3059) 
`generate` code that produces 99% identical summarizations to fairseq on CNN test data, with caching.
 2020-03-02 10:35:53 -05:00
Sam Shleifer
9df74b8bc4
Delete all mentions of Model2Model (#3019) 2020-02-26 11:36:27 -05:00
Lysandre Debut
bb7c468520
Documentation (#2989) 
* All Tokenizers

BertTokenizer + few fixes
RobertaTokenizer
OpenAIGPTTokenizer + Fixes
GPT2Tokenizer + fixes
TransfoXLTokenizer
Correct rst for TransformerXL
XLMTokenizer + fixes
XLNet Tokenizer + Style
DistilBERT + Fix XLNet RST
CTRLTokenizer
CamemBERT Tokenizer
FlaubertTokenizer
XLMRobertaTokenizer
cleanup

* cleanup
 2020-02-25 18:43:36 -05:00
Lysandre Debut
65e7c90a77
Adding usage examples for common tasks (#2850) 
* Usage: Sequence Classification & Question Answering

* Pipeline example

* Language modeling

* TensorFlow code for Sequence classification

* Custom TF/PT toggler in docs

* QA + LM for TensorFlow

* Finish Usage for both PyTorch and TensorFlow

* Addressing Julien's comments

* More assertive

* cleanup

* Favicon
- added favicon option in conf.py along with the favicon image
- udpated 🤗 logo. slightly smaller and should appear more consistent across editing programs (no more tongue on the outside of the mouth)

Co-authored-by: joshchagani <joshua@joshuachagani.com>
 2020-02-25 13:48:24 -05:00
Lysandre
f9ec5ca90b Release: v2.5.1 2020-02-24 18:22:54 -05:00
Sam Shleifer
53ce3854a1
New BartModel (#2745) 
* Results same as fairseq
* Wrote a ton of tests
* Struggled with api signatures
* added some docs
 2020-02-20 18:11:13 -05:00
Lysandre
fb560dcb07 Release: v2.5.0 
Welcome Rust Tokenizers
 2020-02-19 11:46:19 -05:00
Lysandre
fd639e5be3 Correct quickstart example when using the past 2020-02-10 11:25:56 -05:00
Lysandre
dd28830327 Update RoBERTa tips 2020-02-07 16:42:35 -05:00
Lysandre
db97930122 Update XLM-R tips 2020-02-07 16:42:35 -05:00
VictorSanh
ee5a6856ca distilbert-base-cased weights + Readmes + omissions 2020-02-07 15:28:13 -05:00
Julien Chaumond
42f08e596f [examples] rename run_lm_finetuning to run_language_modeling 2020-02-07 09:15:28 -05:00
Julien Chaumond
7748cbbe7d Oopsie 2020-02-06 15:30:02 -05:00
Julien Chaumond
432c12521e [docs] Add menu w/ links to other pages on hf.co 2020-02-06 15:30:02 -05:00
Julien Chaumond
eae8ee0389 [doc] model sharing: mention README.md + tweaks 
cc @lysandrejik @thomwolf
 2020-02-05 14:20:03 -05:00
Lysandre
9c67196b83 Update quickstart 2020-02-04 11:11:37 -05:00
Lysandre
d426b58b9e Patch: v2.4.1 2020-01-31 14:55:33 -05:00
Lysandre
6664ea943d Release: v2.4.0 2020-01-31 09:40:32 -05:00
Hang Le
b43cb09aaa Add layerdrop 2020-01-30 12:05:01 -05:00
Lysandre
93dccf527b Pretrained models 2020-01-30 10:04:18 -05:00
Lysandre
73306d028b FlauBERT documentation 2020-01-30 10:04:18 -05:00
Lysandre
c69b082601 Update documentation 2020-01-29 12:06:13 -05:00
Lysandre
44a5b4bbe7 Update documentation 2020-01-29 11:47:49 -05:00
Wietse de Vries
f5a236c3ca Add Dutch pre-trained BERT model 2020-01-27 21:00:34 -05:00
thomwolf
e0849a66ac adding in the doc 2020-01-27 14:27:07 -05:00
Lysandre
983fef469c AutoModels doc 2020-01-24 16:37:30 -05:00
Lysandre
24d5ad1dcc Run the examples in slow 2020-01-23 09:38:45 -05:00
Lysandre
9ddf60b694 Tips + whitespaces 2020-01-23 09:38:45 -05:00