Uri Alon
a0be960dcc
Fixing tokenizer when transformers is installed without tokenizers ( #26236 )
...
* Fixing tokenizer when tokenizers is not installed
* Adding __repr__ function and repr=True in dataclass
* Revert "Adding __repr__ function and repr=True in dataclass"
This reverts commit 18839505d1
2023-09-27 11:58:04 +02:00
Nour Eddine ZEKAOUI
777f2243f5
Update semantic_segmentation.md ( #26419 )
2023-09-27 11:51:44 +02:00
Shauray Singh
abd2531034
Fix padding for IDEFICS ( #26396 )
...
* fix
* fixup
* tests
* fixup
2023-09-27 10:56:07 +02:00
Nathan Lambert
408b2b3c50
Add torch RMSProp optimizer ( #26425 )
...
add rmsprop
2023-09-26 19:27:09 +02:00
Matt
6ba63ac3a0
[InternLM] Add support for InternLM ( #26302 )
...
* Add config.bias to LLaMA to allow InternLM models to be ported as LLaMA checkpoints
* Rename bias -> attention_bias and add docstring
2023-09-26 16:52:19 +01:00
Hugo Laurençon
0ac3875011
Fix DeepSpeed issue with Idefics ( #26393 )
...
Fix deepspeed issue with Idefics
2023-09-26 10:19:00 +02:00
sanjeevk-os
6ce6a5adb9
added support for gradient checkpointing in ESM models ( #26386 )
2023-09-26 10:15:53 +02:00
titi
a8531f3bfd
Deleted duplicate sentence ( #26394 )
2023-09-26 10:11:28 +02:00
NielsRogge
a09130feee
[ViTMatte] Add resources ( #26317 )
...
Add resource
2023-09-26 07:06:38 +02:00
NielsRogge
ace74d16bd
Add Nougat ( #25942 )
...
* Add conversion script
* Add NougatImageProcessor
* Add crop margin
* More improvements
* Add docs, READMEs
* Remove print statements
* Include model_max_length
* Add NougatTokenizerFast
* Fix imports
* Improve postprocessing
* Improve image processor
* Fix image processor
* Improve normalize method
* More improvements
* More improvements
* Add processor, improve docs
* Simplify fast tokenizer
* Remove test file
* Fix docstrings
* Use NougatProcessor in conversion script
* Add is_levensthein_available
* Add tokenizer tests
* More improvements
* Use numpy instead of opencv
* Add is_cv2_available
* Fix cv2_available
* Add is_nltk_available
* Add image processor tests, improve crop_margin
* Add integration tests
* Improve integration test
* Use do_rescale instead of hacks, thanks Amy
* Remove random_padding
* Address comments
* Address more comments
* Add import
* Address more comments
* Address more comments
* Address comment
* Address comment
* Set max_model_input_sizes
* Add tests
* Add requires_backends
* Add Nougat to exotic tests
* Use to_pil_image
* Address comment regarding nltk
* Add NLTK
* Improve variable names, integration test
* Add test
* refactor, document, and test regexes
* remove named capture groups, add comments
* format
* add non-markdown fixed tokenization
* format
* correct flakyness of args parse
* add regex comments
* test functionalities for crop_image, align long axis and expected output
* add regex tests
* remove cv2 dependency
* test crop_margin equality between cv2 and python
* refactor table regexes to markdown
add newline
* change print to log, improve doc
* fix high count tables correction
* address PR comments: naming, linting, asserts
* Address comments
* Add copied from
* Update conversion script
* Update conversion script to convert both small and base versions
* Add inference example
* Add more info
* Fix style
* Add require annotators to test
* Define all keyword arguments explicitly
* Move cv2 annotator
* Add tokenizer init method
* Transfer checkpoints
* Add reference to Donut
* Address comments
* Skip test
* Remove cv2 method
* Add copied from statements
* Use cached_property
* Fix docstring
* Add file to not doctested
---------
Co-authored-by: Pablo Montalvo <pablo.montalvo.leroux@gmail.com>
2023-09-26 07:06:04 +02:00
Gabriel Yang
5e09af2acd
🌐 [i18n-KO] Translated audio_classification.mdx to Korean ( #26200 )
...
* 🌐 [i18n-KO] Translated to Korean
* update translation
* fix some sentence editing and fixing punctuation
* Update docs/source/ko/_toctree.yml
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
* Apply suggestions from code review
Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>
---------
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
Co-authored-by: Hyeonseo Yun <0525yhs@gmail.com>
2023-09-25 10:24:45 -07:00
qweme32
033ec57c03
Add Russian localization for README ( #26208 )
...
* Add Russian localization
* typo
* mistake in link
* Update README_ru.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update README_ru.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
2023-09-25 09:42:23 -07:00
Yih-Dar
d9e4bc2895
Update tiny model information and pipeline tests ( #26285 )
...
* Update tiny model summary file
* add to pipeline tests
* revert
* fix import
* fix import
* fix
* fix
* update
* update
* update
* fix
* remove BarkModelTest
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-25 18:08:12 +02:00
Maria Khalusova
546e7679e7
[docs] removed MaskFormerSwin and TimmBackbone from the table on index.md ( #26347 )
...
removed MaskFormerSwin and TimmBackbone from the table
2023-09-25 09:41:59 -04:00
Omar Sanseviero
0ee4590684
Fix MusicGen logging error ( #26370 )
...
* Fix logging error
* Update modeling_musicgen.py
* Update modeling_musicgen.py
2023-09-25 13:08:25 +02:00
Nino Risteski
6accd5effb
Update add_new_model.md ( #26365 )
...
fixed typos
2023-09-25 12:58:11 +02:00
HanSeokhyeon
5936c8c57c
Fixed unclosed p tags ( #26240 )
2023-09-22 11:39:28 -07:00
Phuc Van Phan
910faa3e1f
feat: adding num_proc to load_dataset ( #26326 )
...
* feat: adding num_proc to load_dataset
* feat: add add_num_proc for run_mlm_flax
* feat: add num_proc for bart and t5
* chorse: remove
2023-09-22 19:22:47 +02:00
LeviVasconcelos
576cd45a57
Add image to image pipeline ( #25393 )
...
* Add image to image pipeline
Add image to image pipeline
* remove swin2sr from tf auto
* make ImageToImage importable
* make style
make style
make style
make style
* remove tf support
* remove nonused imports
* fix postprocessing
* add important comments; add unit tests
* add documentation
* remove support for TF
* make fixup
* fix typehint Image.Image
* fix documentation code
* address review request; fix unittest type checking
* address review request; fix unittest type checking
* make fixup
* address reviews
* Update src/transformers/pipelines/image_to_image.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* enhance docs
* make style
* make style
* improve docetest time
* improve docetest time
* Update tests/pipelines/test_pipelines_image_to_image.py
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
* Update tests/pipelines/test_pipelines_image_to_image.py
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
* make fixup
* undo faulty merge
* undo faulty merge
* add image-to-image to test pipeline mixin
* Update src/transformers/pipelines/image_to_image.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update tests/pipelines/test_pipelines_image_to_image.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* improve docs
---------
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-09-22 19:53:55 +03:00
Sanchit Gandhi
914771cbfe
[TTA Pipeline] Fix MusicGen test ( #26348 )
...
* fix musicgen pipeline test
* fix wav2vec2 doctest
* revert wav2vec2
2023-09-22 17:55:54 +02:00
Younes Belkada
368a58e61c
[core ] Integrate Flash attention 2 in most used models ( #25598 )
...
* v1
* oops
* working v1
* fixup
* add some TODOs
* fixup
* padding support + try with module replacement
* nit
* alternative design
* oops
* add `use_cache` support for llama
* v1 falcon
* nit
* a bit of refactor
* nit
* nits nits
* add v1 padding support falcon (even though it seemed to work before)
* nit
* falcon works
* fixup
* v1 tests
* nit
* fix generation llama flash
* update tests
* fix tests + nits
* fix copies
* fix nit
* test- padding mask
* stype
* add more mem efficient support
* Update src/transformers/modeling_utils.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* fixup
* nit
* fixup
* remove it from config when saving
* fixup
* revert docstring
* add more checks
* use values
* oops
* new version
* fixup
* add same trick for falcon
* nit
* add another test
* change tests
* fix issues with GC and also falcon
* fixup
* oops
* Update src/transformers/models/falcon/modeling_falcon.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* add init_rope
* updates
* fix copies
* fixup
* fixup
* more clarification
* fixup
* right padding tests
* add docs
* add FA in docker image
* more clarifications
* add some figures
* add todo
* rectify comment
* Change to FA2
* Update docs/source/en/perf_infer_gpu_one.md
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* split in two lines
* change test name
* add more tests
* some clean up
* remove `rearrange` deps
* add more docs
* revert changes on dockerfile
* Revert "revert changes on dockerfile"
This reverts commit 8d72a66b4b
2023-09-22 17:42:10 +02:00
Maria Khalusova
dcbfd93d7a
[doc] fixed indices in obj detection example ( #26343 )
...
fixed indexes in obj detection example
2023-09-22 10:29:27 -04:00
Yih-Dar
c3ecf2d95d
Fix doctest CI ( #26324 )
...
fix doc CI
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-22 08:58:30 +02:00
Yih-Dar
06ee91aebc
Use CircleCI store_test_results ( #26223 )
...
store_test_results
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-22 08:56:54 +02:00
Gema Parreño
587b7b16ce
[QUICK FIX LINK] Update trainer.py ( #26293 )
...
* Update trainer.py
Fix link
* Update src/transformers/trainer.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update trainer.py
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-09-22 03:33:29 +02:00
Matt
000e52aec8
More error message fixup, plus some linebreaks! ( #26296 )
...
* More error message fixup, plus some linebreaks!
* Update src/transformers/dynamic_module_utils.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/dynamic_module_utils.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/dynamic_module_utils.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-09-21 17:36:05 +01:00
Yoach Lacombe
9a30753485
Porting the torchaudio kaldi fbank implementation to audio_utils ( #26182 )
...
* add kaldi fbank
* make style
* add herz_to_mel_kaldi tests
* add mel to hertz kaldi test
* integration tests
* correct test and remove comment
* make style
* Apply suggestions from code review
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
* change parameter name
* Apply suggestions from Arthur review
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update remove_dc_offset description
* fix bug + make style
* fix error in using np.exp instead of np.power
* make style
---------
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-09-21 17:52:47 +02:00
Arthur
b132c1703e
update hf hub dependency to be compatible with the new tokenizers ( #26301 )
2023-09-21 14:57:36 +02:00
Lysandre Debut
26ba56ccbd
Fix FSMT weight sharing ( #26292 )
2023-09-21 14:46:05 +02:00
fxmarty
da971b2271
Keep relevant weights in fp32 when model._keep_in_fp32_modules is set even when accelerate is not installed ( #26225 )
...
* fix bug where weight would not be kept in fp32
* nit
* address review comments
* fix test
2023-09-21 19:00:03 +09:00
Shijie Wu
e3a4bd2bee
add custom RMSNorm to ALL_LAYERNORM_LAYERS ( #26227 )
...
* add LlamaRMSNorm to ALL_LAYERNORM_LAYERS
* fixup
* add IdeficsRMSNorm to ALL_LAYERNORM_LAYERS and fixup
2023-09-20 18:51:56 +02:00
Younes Belkada
0b5024ce72
[Trainer] Refactor trainer + bnb logic ( #26248 )
...
* refactor trainer + bnb logic
* remove logger.info
* oops
2023-09-20 17:38:59 +02:00
Arthur
f94c9b3d86
include changes from llama ( #26260 )
...
* include changes from llama
* add a test
2023-09-20 17:19:30 +02:00
Jinho Park
00247ea0de
add bbox input validation ( #26294 )
2023-09-20 16:48:35 +02:00
fxmarty
245532065d
fix deepspeed available detection ( #26252 )
2023-09-20 16:40:14 +02:00
Matt
f29fe74589
Rewrite for custom code warning messages ( #26291 )
...
Quick britpicking for some warning messages!
2023-09-20 15:18:49 +01:00
Funtowicz Morgan
2d71307dc0
Integrate AMD GPU in CI/CD environment ( #26007 )
...
* Add a Dockerfile for PyTorch + ROCm based on official AMD released artifact
* Add a new artifact single-amdgpu testing on main
* Attempt to test the workflow without merging.
* Changed BERT to check if things are triggered
* Meet the dependencies graph on workflow
* Revert BERT changes
* Add check_runners_amdgpu to correctly mount and check availability
* Rename setup to setup_gpu for CUDA and add setup_amdgpu for AMD
* Fix all the needs.setup -> needs.setup_[gpu|amdgpu] dependencies
* Fix setup dependency graph to use check_runner_amdgpu
* Let's do the runner status check only on AMDGPU target
* Update the Dockerfile.amd to put ourselves in / rather than /var/lib
* Restore the whole setup for CUDA too.
* Let's redisable them
* Change BERT to trigger tests
* Restore BERT
* Add torchaudio with rocm 5.6 to AMD Dockerfile (#26050 )
fix dockerfile
Co-authored-by: Felix Marty <felix@hf.co>
* Place AMD GPU tests in a separate workflow (correct branch) (#26105 )
AMDGPU CI lives in an other workflow
* Fix invalid job name is dependencies.
* Remove tests multi-amdgpu for now.
* Use single-amdgpu
* Use --net=host for now.
* Remote host networking.
* Removed duplicated check_runners_amdgpu step
* Let's tag machine-types with mi210 for now.
* Machine type should be only mi210
* Remove unnecessary push.branches item
* Apply review suggestions moving from `x-amdgpu` to `x-gpu` introducing `amd-gpu` and `miXXX` labels.
* Remove amdgpu from step names.
* finalize
* delete
---------
Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com>
Co-authored-by: Felix Marty <felix@hf.co>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-20 14:48:49 +02:00
Jinho Park
37c205eb5d
Update bros checkpoint ( #26277 )
...
* fix bros integration test
* update bros checkpoint
2023-09-20 10:22:07 +02:00
Sourab Mangrulkar
86ffd5ffa2
fix name error when accelerate is not available ( #26278 )
...
* fix name error when accelerate is not available
* fix `is_fsdp_available`
2023-09-20 08:02:55 +02:00
Sourab Mangrulkar
382ba670ed
FSDP tests and checkpointing fixes ( #26180 )
...
* add fsdp tests
* Update test_fsdp.py
* Update test_fsdp.py
* fixes
* checks
* Update trainer.py
* fix
* fixes for saving/resuming checkpoints
* fixes
* add tests and delete debug statements
* fixing tests
* Update test_fsdp.py
* fix tests
* fix tests
* minor nits
* fix code style and quality
* refactor and modularize test code
* reduce the time of tests
* reduce the test time
* fix test
* reduce test time
* reduce test time
* fix failing tests
* fix
* Apply suggestions from code review
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* resolve comments
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-09-20 10:26:16 +05:30
Sam Passaglia
8e3980a290
[FIX] resize_token_embeddings ( #26102 )
...
* fix roundup command
* add test for resize_token_embeddings
* Update tests/test_modeling_common.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* style
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-09-19 21:44:41 +02:00
Sourab Mangrulkar
ffbf989f0d
DeepSpeed ZeRO-3 handling when resizing embedding layers ( #26259 )
...
* fix failing deepspeed slow tests
* fixes
2023-09-20 00:34:56 +05:30
Yih-Dar
39df4eca73
Fix Error not captured in PR doctesting ( #26215 )
...
* fix
* fix
* fix
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-19 17:27:51 +02:00
NielsRogge
7d6354e047
Add ViTMatte ( #25843 )
...
* First draft
* Simplify image processor
* Fix rebase
* Address comments
* Address more comments
* Address more comments
* Address more comments
* Address more comments
* Improve pad_image
* Add tests
* Update integration test
* Fix image processor tests
* Fix model tests
* Convert checkpoints
* Fix doc tests
* Remove file
* Apply suggestions
* Address comments
* Fix typing hint
* Add batch_norm_eps
* Address comments
* Fix style
2023-09-19 10:56:10 -03:00
Lucain
04191ea1e6
Fix gated repo tests ( #26257 )
...
* Fix gated repo tests
* Apply suggestions from code review
2023-09-19 13:25:12 +02:00
Yih-Dar
eb8489971a
Fix some docstring in image processors ( #26235 )
...
Fix doc
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-19 07:35:41 +02:00
Ralf Müller-Zimmermann
e469be3406
Fix the gitlab user mention in issue templates to the correct user ( #26237 )
2023-09-19 01:49:03 +02:00
Aleksandar Ivanovski
373d0d9985
[docs] Fix model reference in zero shot image classification example ( #26206 )
2023-09-19 00:45:12 +02:00
Nino Risteski
500dfb5b03
Update add_new_pipeline.md ( #26197 )
...
fixed a few typos
2023-09-19 00:41:16 +02:00
Nino Risteski
7d4e0c23c8
Update README.md ( #26198 )
...
Fixed a few typos
2023-09-19 00:02:50 +02:00
