riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-03 03:31:05 +06:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
8,231 Commits 892 Branches 232 Tags 2.1 GiB
be236361f1
Commit Graph

8231 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas Patry
be236361f1
Adding batch_size support for (almost) all pipelines (#13724) 
* Tentative enabling of `batch_size` for pipelines.

* Add systematic test for pipeline batching.

* Enabling batch_size on almost all pipelines

- Not `zero-shot` (it's already passing stuff as batched so trickier)
- Not `QA` (preprocess uses squad features, we need to switch to real
tensors at this boundary.

* Adding `min_length_for_response` for conversational.

* Making CTC, speech mappings avaiable regardless of framework.

* Attempt at fixing automatic tests (ffmpeg not enabled for fast tests)

* Removing ffmpeg dependency in tests.

* Small fixes.

* Slight cleanup.

* Adding docs

and adressing comments.

* Quality.

* Update docs/source/main_classes/pipelines.rst

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/pipelines/question_answering.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/pipelines/zero_shot_classification.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Improving docs.

* Update docs/source/main_classes/pipelines.rst

Co-authored-by: Philipp Schmid <32632186+philschmid@users.noreply.github.com>

* N -> oberved_batch_size

softmax trick.

* Follow `padding_side`.

* Supporting image pipeline batching (and padding).

* Rename `unbatch` -> `loader_batch`.

* unbatch_size forgot.

* Custom padding for offset mappings.

* Attempt to remove librosa.

* Adding require_audio.

* torchaudio.

* Back to using datasets librosa.

* Adding help to set a pad_token on the tokenizer.

* Update src/transformers/pipelines/base.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/pipelines/base.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/pipelines/base.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Quality.

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Philipp Schmid <32632186+philschmid@users.noreply.github.com>
 2021-10-29 11:34:18 +02:00
David del Río Medina
4469010c1b
Replace assertions with RuntimeError exceptions (#14186) 2021-10-28 17:17:43 -04:00
Patrick von Platen
ba71f1b57f
Update README.md 2021-10-28 19:43:05 +02:00
Lysandre
b8fad022a0 v4.13.0.dev0 2021-10-28 12:56:46 -04:00
Lysandre
62bf536631 Release v4.12.0 2021-10-28 12:09:49 -04:00
NielsRogge
5f3bf65111
Fix EncoderDecoderModel docs (#14197) 
* Fix docs

* Apply suggestions from review + fix bug
 2021-10-28 18:01:00 +02:00
NielsRogge
ac12a5ae47
Fix EncoderDecoderModel classes to be more like BART and T5 (#14139) 
* First draft

* Make tuple output more readable

* Replace assertions by value errors

* Make it possible to predict_with_generate for vision and speech models

* Adapt Seq2SeqTrainer to work with VisionEncoderDecoder/SpeechEncoderDecoder

* Add deprecation warning

* Add copied from statements to vision and speech encoder decoders

* Fix failing test

* Apply @patrickvonplaten's suggestion

* Use reshape instead of view for consistency
 2021-10-28 15:29:04 +02:00
Anton Lozhkov
1251072f46
Fix SEW-D implementation differences (#14191) 
* Fix SEW-D

* Update tests

* isort
 2021-10-28 16:22:18 +03:00
Anton Lozhkov
78b6a2ecbd
Add audio-classification benchmarking results (#14192) 2021-10-28 15:59:18 +03:00
NielsRogge
1dc96a760d
Add SegFormer (#14019) 
* First draft

* Make style & quality

* Improve conversion script

* Add print statement to see actual slice

* Make absolute tolerance smaller

* Fix image classification models

* Add post_process_semantic method

* Disable padding

* Improve conversion script

* Rename to ForSemanticSegmentation, add integration test, remove post_process methods

* Improve docs

* Fix code quality

* Fix feature extractor tests

* Fix tests for image classification model

* Delete file

* Add is_torch_available to feature extractor

* Improve documentation of feature extractor methods

* Apply suggestions from @sgugger's code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Apply some more suggestions of code review

* Rebase with master

* Fix rebase issues

* Make sure model only outputs hidden states when the user wants to

* Apply suggestions from code review

* Add pad method

* Support padding of 2d images

* Add print statement

* Add print statement

* Move padding method to SegformerFeatureExtractor

* Fix issue

* Add casting of segmentation maps

* Add test for padding

* Add small note about padding

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-10-28 08:23:52 -04:00
Stas Bekman
123cce6ffc
[modeling_utils] respect original dtype in _get_resized_lm_head (#14181) 
* respect dtype in _get_resized_lm_head

* Update src/transformers/modeling_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* consistency

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-10-27 19:01:50 -07:00
Patrick von Platen
88cd82e801
Update README.md 2021-10-28 02:35:01 +02:00
Patrick von Platen
e118db15d6
Update README.md 2021-10-28 01:59:27 +02:00
Patrick von Platen
01b1466983
[TPU tests] Enable first TPU examples pytorch (#14121) 
* up

* up

* fix

* up

* Update examples/pytorch/test_xla_examples.py

* correct labels

* up

* up

* up

* up

* up

* up
 2021-10-28 01:22:28 +02:00
Anton Lozhkov
232822f36d
Add DistilHuBERT (#14174) 
* Add conversion

* Rename

* Add an integration test and remove layer_norm

* Remove layer_norm from the converter

* wording

* Fix imports
 2021-10-27 20:17:31 +03:00
Lahfa Samy
e5b8ffb848
Replace assert of data/data_collator.py by ValueError (#14131) 
* Replace assert of data_collator.py by ValueError

* Replace assert of data_collator.py by ValueError
 2021-10-27 12:19:10 -04:00
Anton Lozhkov
25ceb81871
[Pipelines] Fix ASR model types check (#14178) 2021-10-27 17:17:47 +03:00
Patrick von Platen
6200fd7bbc
[Gradient checkpointing] Enable for Deberta + DebertaV2 + SEW-D (#14175) 
* up

* up

* finish

* up

* final changes
 2021-10-27 15:47:20 +02:00
Anton Lozhkov
e1dc5afd28
Add SEW CTC models (#14158) 
* Add SEW CTC models

* Update paths

* Update paths
 2021-10-27 12:21:09 +03:00
Lysandre Debut
1e53faeb2e
Fix gelu test for torch 1.10 (#14167) 2021-10-26 22:20:51 -04:00
Kamal Raj
8ddbfe9752
switch to inference_mode from no_gard (#13667) 
* switch to inference_mode from no_gard
faster inference

* added switch to support older version of pytorch
 2021-10-26 18:02:58 -04:00
Emanuel Huber
ebd48c6de5
Replace assertions with ValueError exception (#14142) 
Updated masked-language modeling examples in pytorch
with convention defined by #12789
 2021-10-26 17:14:29 -04:00
Matthew Goldey
42bfb83d74
fix typos in error messages in speech recognition example and modelcard.py (#14166) 
* specify the text column name in the error message

* pluralize the word fields
 2021-10-26 16:36:26 -04:00
Jangwon Park
41dad89f70
chore: typo on ner accelerate example code (#14150) 2021-10-26 16:23:41 -04:00
Lysandre
27c888db6c Fix copies 2021-10-26 15:48:28 -04:00
Jay Zhang
3f23634a17
[ONNX] Add symbolic function for XSoftmax op for exporting to ONNX. (#14013) 
* Add symbolic function for XSoftmax op for exporting to ONNX.

* Fix format issues.

* Fix a CI issue relative to copies.
 2021-10-26 15:25:02 -04:00
Patrick von Platen
9f3aa46f45
Add Unispeech & Unispeech-SAT (#13963) 
* unispeech

* add copy from

* remove hubert copy from

* finish for today

* add unispeech-sat

* adapt more

* up

* up

* up

* up

* add modeling

* add tests

* up

* up

* finish

* up

* Apply suggestions from code review

* up

* up

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* up

* up

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-10-26 18:59:58 +02:00
Patrick von Platen
9799f4e150
Update README.md 2021-10-26 18:59:25 +02:00
Stas Bekman
bfd8176636
[megatron_gpt2] dynamic gelu, add tokenizer, save config (#13928) 
* [megatron_gpt2] dynamic gelu, add tokenizer, save config

* cleanup

* Update src/transformers/models/megatron_gpt2/convert_megatron_gpt2_checkpoint.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* apply suggestions

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-10-26 09:09:54 -07:00
Sergio Valcarcel Macua
919a964b8f
Include Keras tensor in the allowed types (#14155) 
* Include KerasTensor in allowed types

- This allows propagating symbolic tensors through TFBert models and layers' call(),
  which allows converting the subclass models to functional models.

* Style pass

Co-authored-by: Sergio Valcarcel Macua <sergiov@graphcore.ai>
Co-authored-by: matt <rocketknight1@gmail.com>
 2021-10-26 15:08:59 +01:00
Patrick von Platen
f5ed19f57d
[Speech Recognition] - Distributed training: Make sure vocab file removal and creation don't interfer (#14161) 
* up

* better
 2021-10-26 15:59:33 +02:00
Yih-Dar
840fc8dbca
Add vision_encoder_decoder to models/__init__.py (#14151) 
* Add vision_encoder_decoder

* Update _ignore_modules in get_model_modules()

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2021-10-26 07:36:17 -04:00
Patrick von Platen
e248e9b042
up (#14154) 2021-10-26 13:08:18 +02:00
Thomas Chaigneau
1f60df81b2
Add Camembert to models exportable with ONNX (#14059) 
Add Camembert to models exportable with ONNX

Co-authored-by: Thomas.Chaigneau <thomas.chaigneau@arkea.com>
Co-authored-by: Michael Benayoun <mickbenayoun@gmail.com>
 2021-10-26 11:22:22 +02:00
Patrick von Platen
0c3174c758
Add TF<>PT and Flax<>PT everywhere (#14047) 
* up

* up

* up

* up

* up

* up

* up

* add clip

* fix clip PyTorch

* fix clip PyTorch

* up

* up

* up

* up

* up

* up

* up
 2021-10-25 23:55:08 +02:00
Sylvain Gugger
8560b55b5e
Fix lazy init to stop hiding errors in import (#14124) 2021-10-25 16:53:47 -04:00
Patrick von Platen
c99a2832ed
Update README.md 2021-10-25 19:50:36 +02:00
Patrick von Platen
1a9381c60d
Update README.md 2021-10-25 19:49:51 +02:00
Matt
3e8761ab80
Enable DefaultDataCollator class (#14141) 2021-10-25 15:04:54 +01:00
Matt
84b9579da7
Remove unneeded to_tensor() in TF inline example (#14140) 2021-10-25 15:04:36 +01:00
Chi-Liang, Liu
1967c43eb9
BartEnocder add set_input_embeddings (#13960) 
* BartEnocder add set_input_embeddings

To unify the interface, add set_input_embeddings to BartEncoder.

* BartEnocder add get_input_embeddings
 2021-10-25 13:58:29 +02:00
Reza Gharibi
3e04a41a9b
Fix some writing issues in the docs (#14136) 
* Fix some writing issues in the docs

* Run code quality check
 2021-10-25 07:48:02 -04:00
Reza Gharibi
2ac65551ea
Fix rendering of examples version links (#14134) 2021-10-25 07:45:44 -04:00
karthikrangasai
1b871e091b
Supporting Seq2Seq model for question answering task (#13432) 
* Add seq2seq example for QnA on SQuAD Dataset.

* Changes from review - Fixing styling mistakes.

* Added how to example in README, simplified the access to dataset's preprocess function.

* Added tests for the seq2seq QA example.

* Change dataset column name to fix tests.

* Fix test command mistake.

* Add missing argument 'ignore_pad_token_for_loss' from DataTrainingArguments.

* Add missing argument 'num_beams' from DataTrainingArguments.

* Fix processing of output predicted token ids so that tokenizer decode gets appropriate input. Updated assertion conditions on the tests.
 2021-10-25 07:42:53 -04:00
Reza Gharibi
6b83090e80
Fix some typos in the docs (#14126) 
* Fix some typos in the docs

* Fix a styling issue

* Fix code quality check error
 2021-10-25 07:40:44 -04:00
Kevin Ko
95bab53868
Update TP parallel GEMM image (#14112) 
* Update TP parallel GEMM image

* Delete parallelism-tp-parallel_gemm.png

* Update parallelism-tp-parallel_gemm.png
 2021-10-22 12:57:48 -07:00
Li-Huai (Allan) Lin
62ccbe0960
Rename variables with unclear naming (#14122) 
* Rename var

* Add comments
 2021-10-22 19:05:45 +02:00
Antonio Carlos Falcão Petri
05a2afc252
Add missing --validation_split_percentage data args (#14119) 2021-10-22 19:04:54 +02:00
Baizhou Huang
c7ccb2e779
Fix assertion in models (#14090) 
* replace assertions in src/transformers/models/luke/convert_luke_original_pytorch_checkpoint_to_pytorch.py

* replace assertions in src/transformers/models/marian/convert_marian_to_pytorch.py

* Update src/transformers/models/luke/convert_luke_original_pytorch_checkpoint_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/marian/convert_marian_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/marian/convert_marian_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/marian/convert_marian_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/marian/convert_marian_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/marian/convert_marian_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/marian/convert_marian_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/marian/convert_marian_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/marian/convert_marian_to_pytorch.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: skpig <1900012999@pku.edu.cn>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-10-22 10:03:09 -04:00
Sylvain Gugger
16d7b70b80 Update Korean README to master 2021-10-22 08:13:04 -04:00