Sylvain Gugger
d12ceb48ba
Tokenization tutorial ( #5257 )
...
* All done
* Link to the tutorial
* Typo fixes
Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
* Add metnion of the return_xxx args
Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
2020-06-24 18:43:20 -04:00
Sylvain Gugger
6894b486d0
Fix version controller links (for realsies) ( #5251 )
2020-06-24 12:13:43 -04:00
Sylvain Gugger
609e0c583f
Fix links ( #5248 )
2020-06-24 11:35:55 -04:00
Sylvain Gugger
7c41057d50
Add hugs ( #5225 )
2020-06-24 07:56:14 -04:00
Sylvain Gugger
173528e368
Add version control menu ( #5222 )
...
* Add version control menu
* Constify things
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* Apply suggestions from code review
Co-authored-by: Julien Chaumond <chaumond@gmail.com>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Julien Chaumond <chaumond@gmail.com>
2020-06-23 17:05:12 -04:00
Sylvain Gugger
417e492f1e
Quick tour ( #5145 )
...
* Quicktour part 1
* Update
* All done
* Typos
Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
* Address comments in quick tour
* Update docs/source/quicktour.rst
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* Update from feedback
Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
2020-06-22 16:08:09 -04:00
Sylvain Gugger
1262495a91
Add TF auto model to the docs + fix sphinx warnings ( #5187 )
2020-06-22 14:43:52 -04:00
Sylvain Gugger
eb0ca71ef6
Update glossary ( #5148 )
...
* Update glossary
* Update docs/source/glossary.rst
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2020-06-22 08:30:49 -04:00
Vasily Shamporov
9a3f91088c
Add MobileBert ( #4901 )
...
* Add MobileBert
* Quality + Conversion script
* style
* Update src/transformers/modeling_mobilebert.py
* Links to S3
* Style
* TFMobileBert
Slight fixes to the pytorch MobileBert
Style
* MobileBertForMaskedLM (PT + TF)
* MobileBertForNextSentencePrediction (PT + TF)
* MobileFor{MultipleChoice, TokenClassification} (PT + TF)
ss
* Tests + Auto
* Doc
* Tests
* Addressing @sgugger's comments
* Adressing @patrickvonplaten's comments
* Style
* Style
* Integration test
* style
* Model card
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
2020-06-19 16:38:36 -04:00
Suraj Patil
18177a1a60
lm_labels => labels ( #5080 )
2020-06-18 09:16:29 +02:00
Sylvain Gugger
204ebc25e6
Update installation page and add contributing to the doc ( #5084 )
...
* Update installation page and add contributing to the doc
* Remove mention of symlinks
2020-06-17 14:01:10 -04:00
Sylvain Gugger
7291ea0bff
Reorganize documentation ( #5064 )
...
* Reorganize topics and add all models
2020-06-17 07:55:20 -04:00
Sylvain Gugger
011cc0be51
Fix all sphynx warnings ( #5068 )
2020-06-16 16:50:02 -04:00
Yacine Jernite
49c5202522
Eli5 examples ( #4968 )
...
* add eli5 examples
* add dense query script
* query_di
* merging
* merging
* add_utils
* adds nearest neighbor wikipedia
* batch queries
* training_retriever
* new notebooks
* moved retriever traiing script
* finished wiki40b
* max_len_fix
* train_s2s
* retriever_batch_checkpointing
* cleanup
* merge
* dim_fix
* fix_indexer
* fix_wiki40b_snippets
* fix_embed_for_r
* fp32 index
* fix_sparse_q
* joint_training
* remove obsolete datasets
* add_passage_nn_results
* add_passage_nn_results
* add_batch_nn
* add_batch_nn
* add_data_scripts
* notebook
* notebook
* notebook
* fix_multi_gpu
* add_app
* full_caching
* full_caching
* notebook
* sparse_done
* images
* notebook
* add_image_gif
* with_Gif
* add_contr_image
* notebook
* notebook
* notebook
* train_functions
* notebook
* min_retrieval_length
* pandas_option
* notebook
* min_retrieval_length
* notebook
* notebook
* eval_Retriever
* notebook
* images
* notebook
* add_example
* add_example
* notebook
* fireworks
* notebook
* notebook
* joe's notebook comments
* app_update
* notebook
* notebook_link
* captions
* notebook
* assing RetriBert model
* add RetriBert to Auto
* change AutoLMHead to AutoSeq2Seq
* notebook downloads from hf models
* style_black
* style_black
* app_update
* app_update
* fix_app_update
* style
* style
* isort
* Delete WikiELI5training.ipynb
* Delete evaluate_eli5.py
* Delete WikiELI5explore.ipynb
* Delete ExploreWikiELI5Support.html
* Delete explainlikeimfive.py
* Delete wiki_snippets.py
* children before parent
* children before parent
* style_black
* style_black_only
* isort
* isort_new
* Update src/transformers/modeling_retribert.py
Co-authored-by: Julien Chaumond <chaumond@gmail.com>
* typo fixes
* app_without_asset
* cleanup
* Delete ELI5animation.gif
* Delete ELI5contrastive.svg
* Delete ELI5wiki_index.svg
* Delete choco_bis.svg
* Delete fireworks.gif
* Delete huggingface_logo.jpg
* Delete huggingface_logo.svg
* Delete Long_Form_Question_Answering_with_ELI5_and_Wikipedia.ipynb
* Delete eli5_app.py
* Delete eli5_utils.py
* readme
* Update README.md
* unused imports
* moved_info
* default_beam
* ftuned model
* disclaimer
* Update src/transformers/modeling_retribert.py
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* black
* add_doc
* names
* isort_Examples
* isort_Examples
* Add doc to index
Co-authored-by: Julien Chaumond <chaumond@gmail.com>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
2020-06-16 16:36:58 -04:00
Sylvain Gugger
439aa1d6e9
Remove old section + caching in install ( #5027 )
2020-06-16 13:03:41 -04:00
Sylvain Gugger
f9f8a5312e
Add DistilBertForMultipleChoice ( #5032 )
...
* Add `DistilBertForMultipleChoice`
2020-06-15 18:31:41 -04:00
Anthony MOI
36434220fc
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests ( #4510 )
...
* Use tokenizers pre-tokenized pipeline
* failing pretrokenized test
* Fix is_pretokenized in python
* add pretokenized tests
* style and quality
* better tests for batched pretokenized inputs
* tokenizers clean up - new padding_strategy - split the files
* [HUGE] refactoring tokenizers - padding - truncation - tests
* style and quality
* bump up requied tokenizers version to 0.8.0-rc1
* switched padding/truncation API - simpler better backward compat
* updating tests for custom tokenizers
* style and quality - tests on pad
* fix QA pipeline
* fix backward compatibility for max_length only
* style and quality
* Various cleans up - add verbose
* fix tests
* update docstrings
* Fix tests
* Docs reformatted
* __call__ method documented
Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
2020-06-15 17:12:51 -04:00
Sam Shleifer
a9f1fc6c94
Add bart-base ( #5014 )
2020-06-15 13:29:26 -04:00
Suraj Patil
e93ccb3290
BartForQuestionAnswering ( #4908 )
2020-06-12 15:47:57 -04:00
Sylvain Gugger
538531cde5
Add AlbertForMultipleChoice ( #4959 )
...
* Add AlbertForMultipleChoice
* Make up to date and add all models to common tests
2020-06-12 14:20:19 -04:00
Suraj Patil
ef2dcdccaa
ElectraForQuestionAnswering ( #4913 )
...
* ElectraForQuestionAnswering
* udate __init__
* add test for electra qa model
* add ElectraForQuestionAnswering in auto models
* add ElectraForQuestionAnswering in all_model_classes
* fix outputs, input_ids defaults to None
* add ElectraForQuestionAnswering in docs
* remove commented line
2020-06-10 15:17:52 -04:00
Sylvain Gugger
41a1d27cde
Add XLMRobertaForQuestionAnswering ( #4855 )
...
* Add XLMRobertaForQuestionAnswering
* Formatting
* Make test happy
2020-06-08 21:22:37 -04:00
Sylvain Gugger
37be3786cf
Clean documentation ( #4849 )
...
* Clean documentation
2020-06-08 11:28:19 -04:00
Sylvain Gugger
56d5d160cd
Add model and doc badges ( #4811 )
...
* Add badges for models and docs
2020-06-05 18:45:42 -04:00
Sylvain Gugger
5c0cfc2cf0
Add link to community models ( #4804 )
2020-06-05 15:29:20 -04:00
Sylvain Gugger
fa661ce749
Add model summary ( #4789 )
...
* Add model summary
* Add link to pretrained models
2020-06-05 12:22:50 -04:00
Julien Chaumond
99207bd112
Pipelines: miscellanea of QoL improvements and small features... ( #4632 )
...
* [hf_api] Attach all unknown attributes for future-proof compatibility
* [Pipeline] NerPipeline is really a TokenClassificationPipeline
* modelcard.py: I don't think we need to force the download
* Remove config, tokenizer from SUPPORTED_TASKS as we're moving to one model = one weight + one tokenizer
* FillMaskPipeline: also output token in string form
* TextClassificationPipeline: option to return all scores, not just the argmax
* Update docs/source/main_classes/pipelines.rst
2020-06-03 03:51:31 -04:00
Julien Chaumond
b42586ea56
Fix CI after killing archive maps ( #4724 )
...
* 🐛 Fix model ids for BART and Flaubert
2020-06-02 10:21:09 -04:00
Lysandre
b43c78e5d3
Release: v2.11.0
2020-06-02 09:49:09 -04:00
Julien Chaumond
d4c2cb402d
Kill model archive maps ( #4636 )
...
* Kill model archive maps
* Fixup
* Also kill model_archive_map for MaskedBertPreTrainedModel
* Unhook config_archive_map
* Tokenizers: align with model id changes
* make style && make quality
* Fix CI
2020-06-02 09:39:33 -04:00
Patrick von Platen
56ee2560be
[Longformer] Better handling of global attention mask vs local attention mask ( #4672 )
...
* better api
* improve automatic setting of global attention mask
* fix longformer bug
* fix global attention mask in test
* fix global attn mask flatten
* fix slow tests
* update docstring
* update docs and make more robust
* improve attention mask
2020-05-29 17:58:42 +02:00
Patrick von Platen
9c17256447
[Longformer] Multiple choice for longformer ( #4645 )
...
* add multiple choice for longformer
* add models to docs
* adapt docstring
* add test to longformer
* add longformer for mc in init and modeling auto
* fix tests
2020-05-29 13:46:08 +02:00
Lysandre Debut
6a17688021
per_device instead of per_gpu/error thrown when argument unknown ( #4618 )
...
* per_device instead of per_gpu/error thrown when argument unknown
* [docs] Restore examples.md symlink
* Correct absolute links so that symlink to the doc works correctly
* Update src/transformers/hf_argparser.py
Co-authored-by: Julien Chaumond <chaumond@gmail.com>
* Warning + reorder
* Docs
* Style
* not for squad
Co-authored-by: Julien Chaumond <chaumond@gmail.com>
2020-05-27 11:36:55 -04:00
Patrick von Platen
c589eae2b8
[Longformer For Question Answering] Conversion script, doc, small fixes ( #4593 )
...
* add new longformer for question answering model
* add new config as well
* fix links
* fix links part 2
2020-05-26 14:58:47 +02:00
Patrick von Platen
3e3e552125
[Reformer] fix reformer num buckets ( #4564 )
...
* fix reformer num buckets
* fix
* adapt docs
* set num buckets in config
2020-05-25 16:04:45 -04:00
Alexander Measure
95a26fcf2d
link to paper was broken ( #4526 )
...
changed from https://https://arxiv.org/abs/2001.04451.pdf to https://arxiv.org/abs/2001.04451.pdf
2020-05-22 15:17:09 -04:00
Lysandre
e0db6bbd65
Release: v2.10.0
2020-05-22 10:37:44 -04:00
Patrick von Platen
48c3a70b4e
[Longformer] Docs and clean API ( #4464 )
...
* add longformer docs
* improve docs
2020-05-19 21:52:36 +02:00
Iz Beltagy
8f1d047148
Longformer ( #4352 )
...
* first commit
* bug fixes
* better examples
* undo padding
* remove wrong VOCAB_FILES_NAMES
* License
* make style
* make isort happy
* unit tests
* integration test
* make `black` happy by undoing `isort` changes!!
* lint
* no need for the padding value
* batch_size not bsz
* remove unused type casting
* seqlen not seq_len
* staticmethod
* `bert` selfattention instead of `n2`
* uint8 instead of bool + lints
* pad inputs_embeds using embeddings not a constant
* black
* unit test with padding
* fix unit tests
* remove redundant unit test
* upload model weights
* resolve todo
* simpler _mask_invalid_locations without lru_cache + backward compatible masked_fill_
* increase unittest coverage
2020-05-19 16:04:43 +02:00
Soham Chatterjee
fa6113f9a0
Fixed spelling of training ( #4416 )
2020-05-18 11:23:29 -04:00
Lysandre
7cb203fae4
Release: v2.9.1
2020-05-13 17:38:50 -04:00
Sam Shleifer
9a687ebb77
[Marian Fixes] prevent predicting pad_token_id before softmax, support language codes, name multilingual models ( #4290 )
2020-05-13 17:29:41 -04:00
Patrick von Platen
839bfaedb2
[Docs, Notebook] Include generation pipeline ( #4295 )
...
* add first text for generation
* add generation pipeline to usage
* Created using Colaboratory
* correct docstring
* finish
2020-05-13 14:24:08 -04:00
Guo, Quan
39994051e4
Add migrating from pytorch-transformers ( #4273 )
...
"Migrating from pytorch-transformers to transformers" is missing in the main document. It is available in the main `readme` thought. Just move it to the document.
2020-05-11 13:35:13 -04:00
fgaim
41e8291217
Add ALBERT to the Tensorflow to Pytorch model conversion cli ( #3933 )
...
* Add ALBERT to convert command of transformers-cli
* Document ALBERT tf to pytorch model conversion
2020-05-11 13:10:00 -04:00
Stefan Schweter
3f42eb979f
Documentation: fix links to NER examples ( #4279 )
...
* docs: fix link to token classification (NER) example
* examples: fix links to NER scripts
2020-05-11 12:48:21 -04:00
Patrick von Platen
ac7d5f67a2
[Reformer] Add Enwiki8 Reformer Model - Adapt convert script ( #4282 )
...
* adapt convert script
* update convert script
* finish
* fix marian pretrained docs
2020-05-11 16:38:07 +02:00
Sam Shleifer
3487be75ef
[Marian] documentation and AutoModel support ( #4152 )
...
- MarianSentencepieceTokenizer - > MarianTokenizer
- Start using unk token.
- add docs page
- add better generation params to MarianConfig
- more conversion utilities
2020-05-10 13:54:57 -04:00
Girishkumar
9d2f467bfb
[README] Corrected some grammatical mistakes ( #4199 )
2020-05-10 09:02:36 -04:00
Julien Chaumond
c99fe0386b
[doc] Fix broken links + remove crazy big notebook
2020-05-07 18:44:18 -04:00
