Commit Graph

15053 Commits

Author SHA1 Message Date
Yih-Dar
293991d44b
Make add_special_tokens more clear (#20424)
* make add_special_tokens more clear

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-30 12:56:32 +01:00
Yih-Dar
d0c1ded5f3
remove attention_mask truncation in whisper (#20488)
* remove truncation

* For TFWhisper

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-30 11:46:01 +01:00
amyeroberts
de6d19ea92
Add segmentation + object detection image processors (#20160)
* Add transforms for object detection

* DETR models + Yolos

* Scrappy additions

* Maskformer image processor

* Fix up; MaskFormer tests

* Update owlvit processor

* Add to docs

* OwlViT tests

* Update pad logic

* Remove changes to transforms

* Import fn directly

* Update to include pad transformation

* Remove uninstended changes

* Add new owlvit post processing function

* Tidy up

* Fix copies

* Fix some copies

* Include device fix

* Fix scipy imports

* Update _pad_image

* Update padding functionality

* Fix bug

* Properly handle ignore index

* Fix up

* Remove defaults to None in docstrings

* Fix docstrings & docs

* Fix sizes bug

* Resolve conflicts in init

* Cast to float after resizing

* Tidy & add size if missing

* Allow kwards when processing for owlvit

* Update test values
2022-11-30 10:24:03 +00:00
Sanchit Gandhi
ae3cbc9548
[modelcard] Set model name if empty (#20496)
* [modelcard] Set model name if empty

* no magic

Co-authored-by: Sylvain Gugger <sylvain@huggingface.co>

Co-authored-by: Sylvain Gugger <sylvain@huggingface.co>
2022-11-30 09:55:43 +00:00
Sanchit Gandhi
08fad080e3
[modelcard] Check for IterableDataset (#20495) 2022-11-30 09:55:07 +00:00
Sylvain Gugger
ab9fe45236
Fix disk offload for full safetensors checkpoints (#20497) 2022-11-29 14:58:30 -05:00
Francisco Kurucz
4aa630eeab
Fix documentation code to import facebook/detr-resnet-50 model (#20491) 2022-11-29 13:30:26 -05:00
sandeep kumar
86e435bbb1
fixed small typo (#20490)
Co-authored-by: Sandeep Kumar <sandeep.kumar@woven-planet.global>
2022-11-29 11:35:12 -05:00
Billy Lee
73e2faa6c2
Replace assert statements with raise exceptions (#20478)
* replace assert statements with exceptions

* made conditions more readable
2022-11-29 11:34:08 -05:00
Pi Esposito
fb2b45e562
add in layer gpt2 tokenizer (#20421)
* add minimal working gpt2 tokenizer

* graph mode and output equivalence tests working

* not today tensorflow. serialization test passing!

* fix style, documentation, docstrings and all that jazz

* passing consistency checks

* move keras nlp to tf dependencies

* fix tf modeling utils and gpt2 attention to enable compiling

* fix (I hope) keras nlp dependencies

* rever changes on generation

* remove debug prints

* remove redundant tf dummy objects

* add from config, get config and max length settings to address review

* let flake ignore the error on distillation you are welcome

* test from config

* add padding test

* address sgugger review
2022-11-29 10:02:40 -05:00
Yih-Dar
e8d448edcf
extract warnings in GH workflows (#20487)
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-29 15:58:54 +01:00
Sylvain Gugger
bbcd5eea3b
Fix init import_structure sorting (#20477)
* Fix init import_structure sorting

* Fix rebase
2022-11-29 09:46:10 -05:00
fxmarty
3b91f96fc9
Fix torch meshgrid warnings (#20475)
* fix torch meshgrid warnings

* support lower torch versions

* don't edit examples

* dont edit examples

* fix ci

* fix style

* rebase cleanup

* fix ci again
2022-11-29 08:38:23 -05:00
amyeroberts
ae1cffaf3c
Add Donut image processor (#20425)
* Add Donut image processor

* Update src/transformers/image_transforms.py

Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>

* Fix docstrings

* Full var names in docstring

Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
2022-11-29 10:38:01 +00:00
Yih-Dar
28247e7881
Extract warnings from CI artifacts (#20474)
* extract warning from CI artifacts

* fix path

* fix logic

* fix comment

* update default values

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-28 21:14:33 +01:00
NielsRogge
6dc884abc8
[Maskformer] Add MaskFormerSwin backbone (#20344)
* First draft

* Fix backwards compatibility

* More fixes

* More fixes

* Make backbone more general

* Improve backbone

* Improve test

* Fix config checkpoint

* Address comments

* Use model_type

* Address more comments

* Fix special model names

* Remove MaskFormerSwinModel and MaskFormerSwinPreTrainedModel from main init

* Fix typo

* Update backbone

* Apply suggestion

Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
2022-11-28 20:33:49 +01:00
Henghui Zhu
955780d3ab
add timeout option for deepspeed engine (#20443) 2022-11-28 10:23:25 -08:00
Sayak Paul
d59d5a618b
chore: add link to the video cls notebook. (#20386)
* chore: add link to the video cls notebook.

* chore: segregate as resources.
2022-11-28 12:10:24 -05:00
amyeroberts
321ef388fe
Include image processor in add-new-model-like (#20439) 2022-11-28 16:46:02 +00:00
NielsRogge
0bae286de9
[AutoBackbone] Improve API (#20407)
* Add hidden states and attentions to backbone outputs

* Update ResNet

* Fix more tests

* Debug test

* Fix test_determinism

* Fix test_save_load

* Remove file

* Disable fx tests

* Test

* Add fx support for backbones

Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
2022-11-28 17:20:24 +01:00
Arthur
39a72125e7
fix both failing RoCBert tests (#20469) 2022-11-28 17:08:57 +01:00
Sylvain Gugger
30163921ae
Safetensors offload (#20321)
* INtegrate safetensos in weight offloading

* Use safetensors checkpoint for offload when available

* Make naming consistent

* Make load faster

* Quality

* Add default
2022-11-28 10:35:52 -05:00
Lianmin Zheng
ac2f6674a3
[FLAX] Add dtype to embedding for bert/bart/opt/t5 (#20340)
* [FLAX] Add dtype to embedding for bert/bart/opt/t5

* Fix all copies

* Add a test case
2022-11-28 10:21:42 -05:00
Katie Le
667ccea722
Replace assertion with ValueError exceptions in run_image_captioning_flax.py (#20365)
* replace 4 asserts with ValueError exception for control flow

* Update examples/flax/image-captioning/run_image_captioning_flax.py

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* Update examples/flax/image-captioning/run_image_captioning_flax.py

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* reformatted file

* uninstalled trasformers and applied make style

Co-authored-by: Bibi <Bibi@katies-mac.local>
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
2022-11-28 15:06:25 +00:00
sha016
0a6193252e
[Doctest] Add configuration_fsmt.py (#19936)
* fsmt doctest

* Update

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-28 09:47:45 -05:00
JuheonChu
98122794d4
Replace assertions with value errors on distilbert model (#20463)
* Changed assert into 7-8 exceptions

* updated syntax error

* updated error

* updated file (Co-autho: Batese2001)

* Successful test on test_modeling_distilbert.py 

Successful raising errors and exceptions on the revised code in test_modeling_distilbert.py .

Co-credit: @batese2001

* Delete test_modeling_distilbert.ipynb

* Update modeling_distilbert.py

* Successful raising of exceptions with the conditions that are contrary to defined condition that asserts statements (Co-author: Batese2001)

* Successful raising of exceptions with the conditions that are contrary to defined condition that asserts statements (Co-author: Batese2001)

* committing the reformatted distilbert model

* reformatted distilbert model

* reformatted distilbert model

* reformatted distilbert model

* reformatted distilbert model with black

* Changed comments that explain better about raising exceptions for not having the even number of multi heads

* Changed comments that explain better about raising exceptions for not having the even number of multi heads

* changed based on the feedback

* Changed line 833 based on the suggestion made from @younesbelkada

* Changed line 833 based on the suggestion made from @younesbelkada draft2

* reformatted file

* Update src/transformers/models/distilbert/modeling_distilbert.py

* Update src/transformers/models/distilbert/modeling_distilbert.py

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
2022-11-28 09:44:03 -05:00
Patrick von Platen
134a8e21ae
[CLIPTokenizer] Improve warning (#20458) 2022-11-28 15:20:14 +01:00
Wang, Yi
de53e4bf1f
with pytorch cpu only version. without --no_cuda, using --bf16 will trigger error like "Your setup doesn't support bf16/gpu. You need torch>=1.10, using Ampere GPU with cuda>=11.0" (#20445) 2022-11-28 08:56:09 -05:00
Wang, Yi
ca3b652bbd
update cpu related doc (#20444) 2022-11-28 08:54:35 -05:00
LiWenhan
8f7078e822
make tensors in function build_relative_position created on proper device instead of always on cpu (#20434)
Co-authored-by: wenhanli <wenhanli@tencent.com>
2022-11-28 08:45:01 -05:00
Matt
de4159a318
More TF int dtype fixes (#20384)
* Add a test to ensure int dummy inputs are int64

* Move the test into the existing int64 test and update a lot of existing dummies

* Fix remaining dummies

* Fix remaining dummies

* Test for int64 serving sigs as well

* Update core tests to use tf.int64

* Add better messages to the assertions

* Update all serving sigs to int64

* More sneaky hiding tf.int32s

* Add an optional int32 signature in save_pretrained

* make fixup

* Add Amy's suggestions

* Switch all serving sigs back to tf.int32

* Switch all dummies to tf.int32

* Adjust tests to check for tf.int32 instead of tf.int64

* Fix base dummy_inputs dtype

* Start casting to tf.int32 in input_processing

* Change dtype for unpack_inputs test

* Add proper tf.int32 test

* Make the alternate serving signature int64
2022-11-28 13:24:44 +00:00
Matt
72b19ca680
Fix ESM checkpoints for tests (#20436)
* Re-enable TF ESM tests, make sure we use facebook checkpoints

* make fixup
2022-11-28 13:19:28 +00:00
Yih-Dar
f244a97801
Fix doctests for audio models (#20468)
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-28 11:13:34 +01:00
Yih-Dar
df938fc1b4
Fix links for contrastive_loss (#20455)
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-28 11:02:59 +01:00
Yih-Dar
2cdac665b0
Fix device issues in CLIPSegModelIntegrationTest (#20467)
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-28 10:41:28 +01:00
Kamal Raj
61d3928bfb
Fix typo in FSMT Tokenizer (#20456)
* Fix typo

* Update tokenization_fsmt.py
2022-11-25 16:04:01 -08:00
SaulLu
3c39c07f11
fix word_to_tokens docstring format (#20450)
* fix docstring

* fix 2

* add details
2022-11-25 20:28:00 +01:00
Stas Bekman
a547d5bda5
[AnyPrecisionAdamW] test fix (#20454) 2022-11-25 09:02:10 -08:00
Younes Belkada
a1d4563f7a
accelerate support for OwlViT (#20411)
* `accelerate` support for `OwlViT`

- added `accelerate` support
- added slow `fp16` tests

* apply suggestions
2022-11-25 11:20:44 +01:00
Sylvain Gugger
afce73bd9d
Fix ModelOutput instantiation when there is only one tuple (#20416) 2022-11-23 15:09:21 -05:00
fxmarty
993a187c6f
fix device in longformer onnx path (#20419) 2022-11-23 15:07:01 -05:00
Ian C
bc00c29d11
Add Spanish translation of pr_checks.mdx (#20339)
* Update _toctree and clone original doc

* Forgot to translate (lol)

* Translate documentation and update toctree

* Add suggested changes from review
2022-11-23 15:06:29 -05:00
Yih-Dar
9a5b84a007
Use updated model_max_length when saving tokenizers (#20401)
* Use updated values

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-23 18:16:26 +01:00
Younes Belkada
ad654e4484
[BNB] Throw ValueError when trying to cast or assign (#20409)
* `bnb` ValueError when tries to cast or assign

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* remove docstrings

* change error log

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2022-11-23 15:51:50 +01:00
raghavanone
03ae1f060b
change the way sentinel tokens can retrived (#20373)
* change the way sentinel tokens can retrived

* Fix line length for doc string

* Fix line length for doc string

* Add more stronger test for t5 tokenization

* Format file changes

* Make a stronger test for filtering sentinel tokens

* fix file format issues
2022-11-23 09:35:44 -05:00
Younes Belkada
81d82e4f78
fix nasty bnb bug (#20408) 2022-11-23 08:31:08 -05:00
Younes Belkada
658e5d8f58
make daily CI happy (#20410) 2022-11-23 14:24:56 +01:00
Patrick von Platen
81c46679bd
[Image Transformers] to_pil fix float edge cases (#20406)
* Correct type checking

* up
2022-11-23 13:47:59 +01:00
Yih-Dar
1c6309bf79
Fix doctest file path (#20400)
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-11-23 13:40:34 +01:00
Stas Bekman
0ee71188ff
[bloom] convert script tweaks (#18593)
* [bloom] convert script tweaks

* Update src/transformers/models/bloom/convert_bloom_original_checkpoint_to_pytorch.py

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* improve the 2nd assert

* add conversion readme

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
2022-11-22 16:09:43 -08:00