riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity
7,798 Commits 887 Branches 232 Tags 2.1 GiB
a3f96f366a
Commit Graph

7798 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas Patry
a3f96f366a
Moving translation pipeline to new testing scheme. (#13297) 
* Moving `translation` pipeline to new testing scheme.

* Update tokenization mbart tests.
 2021-08-27 12:26:17 +02:00
Stefan Schweter
319d840b46
examples: add keep_linebreaks option to CLM examples (#13150) 
* examples: add keep_linebreaks option to text dataset loader for all CLM examples

* examples: introduce new keep_linebreaks option as data argument in CLM examples
 2021-08-27 11:35:45 +02:00
Nicolas Patry
45a8eb66bb
Moving token-classification pipeline to new testing. (#13286) 
* Moving `token-classification` pipeline to new testing.

* Fix tests.
 2021-08-27 11:24:56 +02:00
Nicolas Patry
a6e36558ef
Moving text-generation pipeline to new testing framework. (#13285) 
* Moving `text-generation` pipeline to new testing framework.

* Keep check_model_type but log instead of raise Exception.

* warning -> error.
 2021-08-26 17:30:03 +02:00
NielsRogge
0759f2510c
Add DINO conversion script (#13265) 
* First commit

* Add interpolation of patch embeddings

* Comment out code

* Fix bug

* Fix another bug

* Fix bug

* Fix another bug

* Remove print statements

* Update conversion script

* Use the official vit implementation

* Add support for converting dino_vits8

* Add DINO to docs of ViT

* Remove assertion

* Add interpolation of position encodings

* Fix bug

* Add align_corners

* Add interpolate_pos_encoding option to forward pass of ViTModel

* Improve interpolate_pos_encoding method

* Add docstring
 2021-08-26 17:25:20 +02:00
Nicolas Patry
14e52783f6
Moving text2text-generation to new pipeline testing mecanism. (#13283) 2021-08-26 16:26:58 +02:00
Nicolas Patry
662b143b71
Hotfixing master tests. (#13282) 2021-08-26 10:09:53 -04:00
Nicolas Patry
59c378d069
Moving text2text-generation to new pipeline testing mecanism. (#13281) 2021-08-26 16:09:48 +02:00
Nicolas Patry
0ebda5382b
Moving table-question-answering pipeline to new testing. (#13280) 2021-08-26 09:09:57 -04:00
Nicolas Patry
879fe8fa75
Moving summarization pipeline to new testing format. (#13279) 
* Moving `summarization` pipeline to new testing format.

* Remove generate_kwargs from __init__ args.
 2021-08-26 14:47:11 +02:00
Nicolas Patry
55fb88d369
Moving question_answering tests to the new testing scheme. Had to tweak a little some ModelTesterConfig for pipelines. (#13277) 
* Moving question_answering tests to the new testing scheme. Had to tweak
a little some ModelTesterConfig for pipelines.

* Removing commented code.
 2021-08-26 12:37:55 +02:00
Nicolas Patry
4fa1cd995c
Fixing the test (warnings was incorrect.) (#13278) 2021-08-26 06:13:48 -04:00
Nicolas Patry
6b586ed18c
Move image-classification pipeline to new testing (#13272) 
- Enforce `test_small_models_{tf,pt}` methods to exist (enforce checking
actual values in small tests)
- Add support for non RGB image for the pipeline.
 2021-08-26 05:52:49 -04:00
Bram Vanroy
401377e679
Add error message concerning revision (#13266) 
* add error message concerning revision

* Update src/transformers/configuration_utils.py

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* re-add double line endings

* is not None instead of implicit bool casting

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
 2021-08-26 04:32:57 -04:00
Stas Bekman
40d60e1536
fix tokenizer_class_from_name for models with - in the name (#13251) 
* fix tokenizer_class_from_name

* Update src/transformers/models/auto/tokenization_auto.py

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* add test

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
 2021-08-26 04:29:14 -04:00
Nicolas Patry
83bfdbdd75
Migrating conversational pipeline tests to new testing format (#13114) 
* New test format for conversational.

* Putting back old mixin.

* Re-enabling auto tests with LazyLoading.

* Feature extraction tests.

* Remove feature-extraction.

* Feature extraction with feature_extractor (No pun intended).

* Update check_model_type for fill-mask.
 2021-08-26 03:50:43 -04:00
Lysandre Debut
72eefb34a9
Add require flax to test (#13260) 2021-08-25 12:56:25 -04:00
Lysandre Debut
5af8df5afb
Some model_types cannot be in the mapping (#13259) 
* Some tokenizers cannot be in the mapping

* Style
 2021-08-25 12:56:16 -04:00
Lysandre Debut
68b6907290
Add CLIP tokenizer to AutoTokenizer (#13258) 2021-08-25 12:56:07 -04:00
Lysandre Debut
3bbe68f837
Hubert test fix (#13261) 2021-08-25 18:41:26 +02:00
Lysandre Debut
3bb4466260
Better notification service (#13267) 2021-08-25 12:14:44 -04:00
Nishant Prabhu
225de5ccbb
Replace assert statement with if condition and ValueError (#13263) 2021-08-25 12:14:03 -04:00
Lysandre
46554fc12f Grad enabled typo 2021-08-25 11:39:45 +02:00
Lysandre Debut
0e4f727069
Remove side effects of disabling gradient computaiton (#13257) 2021-08-25 05:32:51 -04:00
Will Frey
b1198a8440
Update generation_logits_process.py (#12671) 
If you're using type hints, then passing an `int` where a `float` is annotated is acceptable as per [PEP 484](https://www.python.org/dev/peps/pep-0484/#the-numeric-tower).

This makes life a little nicer.
 2021-08-25 02:34:05 +08:00
dependabot[bot]
0245cee469
Bump notebook from 6.1.5 to 6.4.1 in /examples/research_projects/lxmert (#13226) 
Bumps [notebook](http://jupyter.org) from 6.1.5 to 6.4.1.

---
updated-dependencies:
- dependency-name: notebook
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2021-08-24 09:52:39 -04:00
Ambesh Shekhar
0512bfe79e
Custom errors and BatchSizeError (#13184) 
* Adding custom errors and BatchSizeError for GPT2

* Adding custom errors and BatchSizeError for GPT2

* Changing Exception to BaseException

* Exception

* Adding args to Custom Exception

* Adding args to Custom Exception

* Changing from BaseException to Exception

* Changing Conditional loop syntax

* Adding Copyright info

* Handling check_code_quality

* Handling check_code_quality pt2

* Handling check_code_quality pt3

* Handling check_code_quality pt4

* Handling check_code_quality pt5

* Handling check_code_quality pt6

* Handling check_code_quality pt6

* Using black for check_code_quality

* sorting import style

* Changing

* Changing

* verified through style_doc.py

* verified through style_doc.py

* applying isort

* Removing indentation

* Changing

* Changing

* Changing

* Used ValueError

* Using ValueError

* Reformatted Style doc

* Using style doc on modeling_gp2.py

* Adding indentation

* Changing
 2021-08-24 09:01:01 -04:00
Ori Ram
cf57447648
Fix broken links in Splinter documentation (#13237) 2021-08-24 07:55:21 -04:00
Stas Bekman
5c6eca71a9
fix AutoModel.from_pretrained(..., torch_dtype=...) (#13209) 
* fix AutoModel.from_pretrained(..., torch_dtype=...)

* fix to_diff_dict

* add better test

* torch is not always available when a model has self.torch_dtype
 2021-08-24 11:43:41 +02:00
Bram Vanroy
39db2f3c19
Allow local_files_only for fast pretrained tokenizers (#13225) 
* allow local_files_only for fast pretrained tokenizers

* make style
 2021-08-24 03:05:33 -04:00
Lysandre Debut
2772d3e79d
Add RemBert to AutoTokenizer (#13224) 2021-08-23 13:16:48 -04:00
Allan Lin
f1bb6f0839
Fix load tf alias in Albert. (#13159) 2021-08-23 12:08:33 -04:00
Kamal Raj
0b54046ff8
remove unwanted code (#13145) 2021-08-23 12:07:41 -04:00
Yih-Dar
2e20c0f34a
Make Flax GPT2 working with cross attention (#13008) 
* make flax gpt2 working with cross attention

* Remove encoder->decoder projection layer

* A draft (incomplete) for FlaxEncoderDecoderModel

* Add the method from_encoder_decoder_pretrained + the docstrings

* Fix the mistakes of using EncoderDecoderModel

* Fix style

* Add FlaxEncoderDecoderModel to the library

* Fix cyclic imports

* Add FlaxEncoderDecoderModel to modeling_flax_auto.py

* Remove question comments

* add tests for FlaxEncoderDecoderModel

* add flax_encoder_decoder to the lists of ignored entries in check_repo.py

* fix missing required positional arguments

* Remove **kwargs when creating FlaxEncoderDecoderModel in from_encoder_decoder_pretrained()

Also fix generation eos/pad tokens issue

* Fix: Use sequences from the generated_output

* Change a check from assert to raise ValueError

* Fix examples and token ids issues

* Fix missing all_cross_attentions when outputting tuple in modeling_gpt2

* Remove the changes in configuration docstrings.

* allow for bert 2 gpt2

* make fix-copies

* Apply suggestions from code review

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Change remaining examples to bert2gpt2

* Change the test to Bert2GPT2

* Fix examples

* Fix import

* Fix unpack bug

* Rename to FlaxEncoderDecoderModelTest and change the test to bert2gpt2

* Apply suggestions from code review

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Fix: NotImplentedError -> NotImplementedError

* Apply suggestions from code review

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* up

* finalize

Co-authored-by: ydshieh <ydshieh@user.noreply>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
 2021-08-23 17:57:29 +02:00
SaulLu
7223844df9
Change how "additional_special_tokens" argument in the ".from_pretrained" method of the tokenizer is taken into account (#13056) 
* add test

* add change in PretrainedTokenizerBase

* change Luke

* deactivate

* add the possibility to add additional special tokens for M2M100

* format

* add special test for canine

* proposed changes for mbart

* proposed changes for mbart50

* proposed changes for byt5

* proposed changes for canine

* proposed changes for t5

* test fast and slow

* remove comment

* remove comment

* add fast version for all tests

* replace break by continue

* add more comments

* add check to avoid duplicates

* remove comment

* format

* proposed change for wave2vec2

* reverse changes mbart

* uncomment

* format
 2021-08-23 14:35:18 +02:00
sourabh112
b13c6c18d0
correcting group beam search function output score bug (#13211) 2021-08-23 13:27:24 +02:00
Philipp Schmid
f689743e74
SageMaker: Fix sagemaker DDP & metric logs (#13181) 
* Barrier -> barrier

* added logger for metrics

* removed stream handler in trainer

* moved handler

* removed streamhandler from trainer

* updated test image and instance type added datasets version to test

* Update tests/sagemaker/scripts/pytorch/requirements.txt

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
 2021-08-23 10:18:07 +02:00
NielsRogge
8679bd7144
Add min and max question length options to TapasTokenizer (#12803) 
* Add min and max question length option to the tokenizer

* Add corresponding test
 2021-08-23 03:44:42 -04:00
NielsRogge
588e6caa15
Overwrite get_clean_sequence as this was causing a bottleneck (#13183) 2021-08-23 03:41:35 -04:00
StevenTang1998
143738214c
Fix the loss calculation of ProphetNet (#13132) 
* Fix the loss calculation of ProphetNet

* Fix the loss calculation of ProphetNet

Fix the loss calculation of ProphetNet and remove warning
 2021-08-20 11:01:54 +02:00
Allan Lin
91ff480e26
Update namespaces inside torch.utils.data to the latest. (#13167) 
* Update torch.utils.data namespaces to the latest.

* Format

* Update Dataloader.

* Style
 2021-08-19 14:29:51 +02:00
Jannis Vamvas
1fec32adc6
Fix generation docstrings regarding input_ids=None (#12823) 2021-08-18 16:51:54 +02:00
Patrick von Platen
ecfa7eb260
[AutoFeatureExtractor] Fix loading of local folders if config.json exists (#13166) 
* up

* up
 2021-08-18 16:18:13 +02:00
Ori Ram
439a43b6b4
Add splinter (#12955) 
* splinter template

* initialize splinter classes

* Splinter Tokenizer

* splinter.rst

* tokenization fixes

* Documentation & some minor variable name changes

* bug fix (added back question_token_id to config) + variable names

* Minor bug fixes + variable name changes

* Fix Splinter references after merge with new transformers

* changes after running make style & quality

* Fix documentation unindent

* Fix doc indentation in tokenization_splinter

* Fix also SplinterTokenizerFast

* Add Splinter to index.rst and README

* Fixdouble whitespace from index.rst

* Fixed index.rst with 'make fix-copies'

* Update docs/source/model_doc/splinter.rst

Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Update docs/source/model_doc/splinter.rst

Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Update docs/source/model_doc/splinter.rst

Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Update docs/source/model_doc/splinter.rst

Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Update src/transformers/models/splinter/__init__.py

Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Added "copied from BERT" comments

* Removing unnexessary code from modeling_splinter

* Update README.md

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/splinter/configuration_splinter.py

Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Remove references to TF modeling from splinter

* Update src/transformers/models/splinter/modeling_splinter.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Remove unnecessary check

* Update src/transformers/models/splinter/modeling_splinter.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Add differences between Splinter and Bert tokenizers

* Update src/transformers/models/splinter/modeling_splinter.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/splinter/tokenization_splinter_fast.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Remove unnecessary check

* Doc formatting

* Update src/transformers/models/splinter/tokenization_splinter.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/splinter/tokenization_splinter.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* bug fix: remove load_tf_weights attribute

* Some minor quality changes

* Update docs/source/model_doc/splinter.rst

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/transformers/models/splinter/configuration_splinter.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Change FullyConnectedLayer to SplinterFullyConnectedLayer

* Variable naming

* Reove gather_positions function

* Remove ClassificationHead as it's outdated

* Update src/transformers/models/splinter/modeling_splinter.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Remove hardcoded 102 token id

* Minor style change

* Added "tau" organization to all model identifiers & URLS

* Added tau to the tests as well

* Copy-from comments

* Removed all unnecessary classes (e.g. SplinterForMaskedLM)

* Running make fix-copies

* Bug fix: Further removed unnecessary classes

* Add Splinter to AutoTokenization

* Add an integration test for Splinter

* Removed initialize_new_qass from config - It will be done through different checkpoints

* Removed `initialize_new_qass` from documentation as well

* Added new checkpoint names (`tau/splinter-base-qass` and same for large) in the code

* Minor change to test

* SplinterTokenizer now doesn't abstract from BertTokenizer

* SplinterTokenizerFast also dosn't abstract from Bert

* style and quality

* bug fix: import ing torch in tests only if it's available

* Auto mappings

* Changed copyrights in Splinter's files

* Update src/transformers/models/splinter/configuration_splinter.py

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

Co-authored-by: yuvalkirstain <kirstain.yuval@gmail.com>
Co-authored-by: Suraj Patil <surajp815@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
 2021-08-17 08:29:01 -04:00
Nicolas Patry
6626d8a62f
Optimizes ByT5 tokenizer (#13119) 
* Starting to optimize ByT5.

* Making ByT5Tokenizer faster.

* Even faster.

* Cleaning up.
 2021-08-17 10:11:58 +02:00
sararb
14e9d2954c
compute seq_len from inputs_embeds (#13128) 2021-08-16 18:36:08 +02:00
Lysandre Debut
e2f07c01e9
Ci continue through smi failure (#13140) 
* Continue on error

* Specific

* Temporary patch
 2021-08-16 11:40:38 -04:00
Patrick von Platen
73caccde3f
fix bug (#13051) 2021-08-16 16:02:34 +02:00
Omar Sanseviero
c066598c23
Fix frameworks table so it's alphabetical (#13118) 
* Fix frameworks table so it's alphabetical

* Update index.rst

* Don't differentiate when sorting between upper and lower case
 2021-08-16 15:45:19 +02:00
Lysandre
62ba3b6b43 Depend on hidden_dropout_prob 2021-08-16 10:52:28 +02:00