Marc Sun
f54d82cace
[CI] Quantization workflow ( #29046 )
...
* [CI] Quantization workflow
* build dockerfile
* fix dockerfile
* update self-cheduled.yml
* test build dockerfile on push
* fix torch install
* udapte to python 3.10
* update aqlm version
* uncomment build dockerfile
* tests if the scheduler works
* fix docker
* do not trigger on psuh again
* add additional runs
* test again
* all good
* style
* Update .github/workflows/self-scheduled.yml
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* test build dockerfile with torch 2.2.0
* fix extra
* clean
* revert changes
* Revert "revert changes"
This reverts commit 4cb52b8822
2024-02-28 10:09:25 -05:00
RaymondLi0
63caa370e6
Starcoder2 model - bis ( #29215 )
...
* Copy model
* changes
* misc
* fixes
* add embed and residual dropout (#30 )
* misc
* remove rms norm and gated MLP
* remove copied mentions where its not a copy anymore
* remove unused _shape
* copied from mistral instead
* fix copies
* fix copies
* add not doctested
* fix
* fix copyright
* Update docs/source/en/model_doc/starcoder2.md
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/models/starcoder2/configuration_starcoder2.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/models/starcoder2/configuration_starcoder2.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* fix doc
* revert some changes
* add fa2 tests
* fix styling nit
* fix
* push dummy docs
---------
Co-authored-by: Joel Lamy-Poirier <joel.lamy-poirier@servicenow.com>
Co-authored-by: younesbelkada <younesbelkada@gmail.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2024-02-28 01:24:34 +01:00
Michael
83ab0115d1
[i18n-zh] Translate fsdp.md into Chinese ( #29305 )
...
* [i18n-zh] Translate fsdp.md into Chinese
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
* apply suggestions from Fan-Lin
---------
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
2024-02-27 11:26:57 -08:00
Merve Noyan
83e366bfd4
Image Feature Extraction docs ( #28973 )
...
* Image Feature Extraction docs
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update image_feature_extraction.md
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Address comments
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/image_feature_extraction.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update image_feature_extraction.md
* Update image_feature_extraction.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
2024-02-27 09:39:58 +00:00
Eduardo Pacheco
3fcfbe7549
Adding SegGPT ( #27735 )
...
* First commit
* Improvements
* More improvements
* Converted original checkpoint to HF checkpoint
* Fix style
* Fixed forward
* More improvements
* More improvements
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Remove asserts
* Remove unnecessary attributes
* Changed model name to camel case
* Improve forward doc
* Improve tests
* More improvements
* Fix copies
* Fix doc
* Make SegGptImageProcessor more flexible
* Added few-shot test
* Fix style
* Update READMEs and docs
* Update READMEs
* Make inputs required
* Add SegGptForImageSegmentation
* Make tests pass
* Rename to out_indicies
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Fixed naming convention
* Copying SegGptMlp from modeling_sam.py
* Some minor improvements
* Remove mlp_ratio
* Fix docstrings
* Fixed docstring match
* Objects defined before use
* Storing only patch_size and beta for SegGptLoss
* removed _prepare_inputs method
* Removed modified from headers
* Renamed to output_indicies
* Removed unnecessary einsums
* Update tests/models/seggpt/test_modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update tests/models/seggpt/test_modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update tests/models/seggpt/test_modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Fixing issues
* Raise error as soon as possible
* More fixes
* Fix merge
* Added palette to SegGptImageProcessor
* Fixed typo
* Fixed shape typo
* Added permute before doing palette to class mapping
* Fixed style
* Fixed and added tests
* Fixed docstrings
* Matching SegFormer API for post_processing_semantic_segmentation
* Fixed copies
* Fixed SegGptImageProcessor to handle both binary and RGB masks
* Updated docstrings of SegGptImageProcessor
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update docs/source/en/model_doc/seggpt.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/configuration_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/convert_seggpt_to_hf.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update tests/models/seggpt/test_image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update tests/models/seggpt/test_modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Object definitions above & fix style
* Renamed output_indices to intermediate_feature_indices
* Removed unnecessary check on bool_masked_pos
* Loss first in the outputs
* Added validation for do_normalize
* Improved SegGptImageProcessor and added new tests
* Added comment
* Added docstrings to SegGptLoss
* Reimplemented ensemble condition logic in SegGptEncoder
* Update src/transformers/models/seggpt/__init__.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/seggpt/convert_seggpt_to_hf.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/seggpt/configuration_seggpt.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Updated docstrings to use post_process_semantic_segmentation
* Fixed typo on docstrings
* moved pixel values test to test_image_processing_seggpt
* Addressed comments
* Update src/transformers/models/seggpt/configuration_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/image_processing_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/configuration_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Updated docstrings for SegGptLoss
* Address comments
* Added SegGpt example to model docs
* Update src/transformers/models/seggpt/modeling_seggpt.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* moved patchify and unpatchify
* Rename checkpoint
* Renamed intermediate_features to intermediate_hidden_states for consistency
* Update src/transformers/models/seggpt/configuration_seggpt.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Replaced post_process_masks for post_process_semantic_segmentation in the docs
---------
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Niels <niels.rogge1@gmail.com>
Co-authored-by: Eduardo Pacheco <eduardo.pacheco@limehome.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-02-26 18:17:19 +00:00
Michael
a44d2dc3a9
[i18n-zh] Translated task/asr.md into Chinese ( #29233 )
...
* [zh] Translate a task: asr.md
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
* apply suggestions from Fan-Lin
---------
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
2024-02-26 08:53:05 -08:00
Ming Xu (徐明)
734eb25476
🌐 [i18n-ZH] Translate chat_templating.md into Chinese ( #28790 )
...
* [Pix2struct] Simplify generation (#22527 )
* Add model to doc tests
* Remove generate and replace by prepare_inputs_for_generation
* More fixes
* Remove print statements
* Update integration tests
* Fix generate
* Remove model from auto mapping
* Use auto processor
* Fix integration tests
* Fix test
* Add inference code snippet
* Remove is_encoder_decoder
* Update docs
* Remove notebook link
* Release: v4.28.0
* Revert (for now) the change on `Deta` in #22437 (#22750 )
fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
* Patch release: v4.28.1
* update zh chat template.
* Update docs/source/zh/chat_templating.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/zh/_toctree.yml
Co-authored-by: Michael <haifeng.yao@daocloud.io>
* Update docs/source/zh/chat_templating.md
Co-authored-by: Michael <haifeng.yao@daocloud.io>
* Update docs/source/zh/chat_templating.md
Co-authored-by: Michael <haifeng.yao@daocloud.io>
* Update docs/source/zh/chat_templating.md
Co-authored-by: Michael <haifeng.yao@daocloud.io>
* Update docs/source/zh/chat_templating.md
Co-authored-by: Michael <haifeng.yao@daocloud.io>
* Update docs/source/zh/chat_templating.md
Co-authored-by: Michael <haifeng.yao@daocloud.io>
* Update docs/source/zh/chat_templating.md
Co-authored-by: Michael <haifeng.yao@daocloud.io>
---------
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <Sylvain.gugger@gmail.com>
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Michael <haifeng.yao@daocloud.io>
2024-02-26 08:42:24 -08:00
Michael
b43340455d
[i18n-zh] Translated torchscript.md into Chinese ( #29234 )
...
Signed-off-by: windsonsea <haifeng.yao@daocloud.io>
2024-02-26 08:27:47 -08:00
Aaron Jimenez
9f7535bda8
[docs] Spanish translation of tasks_explained.md ( #29224 )
...
* Add tasks_explained.md to es/
* Fix little typo in en/ version
* translate speach/audio section
* translate part of vision computer section | fix little typo in en/
* Fix little typo in en/
* Translate vision computer section | remove ** ** to * * in both files
* Translate NLP section | fix link to task/translation in en/
* Updete link in es/tasks_summary.md
* Fix task_summary title link
2024-02-26 08:18:15 -08:00
Arthur
89c64817ce
[Doc] update model doc qwen2 ( #29238 )
...
* update model doc qwen2
* Update docs/source/en/model_doc/qwen2.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
---------
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
2024-02-23 10:43:31 +01:00
fxmarty
2cc8cf6ce7
Fix torch.compile with fullgraph=True when attention_mask input is used ( #29211 )
...
* fix torch.export.export for llama
* do not change doc title
* make fix copies
2024-02-22 16:40:06 +01:00
NielsRogge
dabe855668
[Mistral, Mixtral] Improve docs ( #29084 )
...
* Improve docs
* Improve chat template
2024-02-22 11:48:01 +01:00
Andrei Panferov
fc37f38915
Add training version check for AQLM quantizer. ( #29142 )
...
* training version check
* warn old aqlm
* aqlm 1.0.2 real
* docs
2024-02-21 17:09:36 +01:00
Arthur
594c1277b2
[ gemma] Adds support for Gemma 💎 ( #29167 )
...
* inital commit
* update
* update conversion checkpoint
* update conversion script
* nits
* some fixes
* nits
* merge
* fix permute
* nits
* fix
* nits
* nits
* nits
* fix rope
* fix both rope
* nites
* style
* make sure flax works
* fix flax init code
* fix foward
* nits
* print flax generation out
* current code
* nits
* SIIIIIIIIIIIIIIIIIII
* update
* add new tokenizer
* correct fast tokenizer
* fix conversion
* more comments
* fix modeling and conversion
* nits and nits
* nits testing
* add some tokenization tests
* add some edge cases
* add slow tests and fix them
* fixup
* fix copies for modeling
* fix copies
* add 7B slow tests
* fix
* fix
* fix tests
* make tokenizer cis go green
* styling
* last tokenizer nits
* update jax tests
* fix flax for 7b
* add jit testing 🤗
* cleanups
* isolated nit, inv_freq for rotary_emb.inv_freq
* propagate to jax
* Apply suggestions from code review
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
* adjust test
* fix conversion script
* change name
* correct file names
* update conversion script
* Fix bos and eos token ids in the model configuration (#3 )
* update modelling
* update conversion script
* add static cache for gemma
* fix sdpa generate
* fix batched
* multiple fixes
* fix FA2
* final fix
* Rename a few missing strings and filenames (#4 )
* merge with upstream main
* fix copies
* fix copies
* fix fixup
* fix fixup
* fix
* fix
* final tests
* fix fx gemma tests
* fix fx bf16/fp16 tests
* update slow fx tests
* fx slow tests: one logits, one generation
* move jit test standalone
* Apply suggestions from code review
* nits
* tokenizer updates
* more tokenization updates: custom GemmaSentencepieceExtrator
* style
* Update src/transformers/cache_utils.py
* Update src/transformers/models/gemma/__init__.py
* Update tests/models/gemma/test_modeling_flax_gemma.py
* small nits
* style
* update tokenization test
* fix the rotary embedding
* with style
* fix slow tests
* WARNING this commit might be very important for precisions
* Update tests/models/gemma/test_modeling_flax_gemma.py
* Update src/transformers/models/gemma/configuration_gemma.py
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Update src/transformers/models/gemma/modeling_flax_gemma.py
Co-authored-by: Lysandre Debut <hi@lysand.re>
* small nits here and there!
* forgotten nit
* remove on the fly computation of inv_freq
* revert previous change, let's be safe and for now re-compute freq cis to make sure it's in float
* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update src/transformers/models/gemma/convert_gemma_weights_to_hf.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update src/transformers/models/gemma/convert_gemma_weights_to_hf.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_flax_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_tokenization_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_tokenization_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_tokenization_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_tokenization_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update tests/models/gemma/test_modeling_gemma.py
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* nit conversion script link
* fix some tests
* add not doctest and pr doctest
* repo consistency
* fix last CIs 🚀
* update all readmes
---------
Co-authored-by: younesbelkada <younesbelkada@gmail.com>
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Co-authored-by: sanchit-gandhi <sanchit@huggingface.co>
Co-authored-by: Lysandre Debut <hi@lysand.re>
2024-02-21 14:21:28 +01:00
Ekaterina Aidova
1d0ea7abe0
support SDPA Attention in stablelm ( #29106 )
...
* support SDPA Attention in stablelm
* add integration test
* add fallback for output_attentions
* Update src/transformers/models/stablelm/modeling_stablelm.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update tests/models/stablelm/test_modeling_stablelm.py
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update src/transformers/models/stablelm/modeling_stablelm.py
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* handle non-contiguous states
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
2024-02-21 13:12:49 +01:00
Gustavo Isturiz
3c00b885b9
Added image_captioning version in es and included in toctree file ( #29104 )
...
added image_captioning version in es and included in toctree file
2024-02-20 09:13:15 -08:00
JB (Don)
b8b16475d4
[Phi] Add support for sdpa ( #29108 )
2024-02-20 14:33:12 +01:00
NielsRogge
07e3454f03
[Docs] Add resources ( #28705 )
...
* Add resource
* Add more resources
* Add resources
* Apply suggestions from code review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Remove mention
* Remove pipeline tags
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-02-19 15:22:29 +01:00
Winton Davies
593230f0a1
fix the post-processing link ( #29091 )
...
The link in evaluation was missing a hyphen between post and processing. I fixed this, for English only. Someone with the ability to do a global search/replace should fix the other languages (if indeed they have this issue)/
2024-02-19 10:15:58 +00:00
Aaron Jimenez
ce4fff0be7
[Docs] Spanish translation of task_summary.md ( #28844 )
...
* Add task_summary to es/_toctree.yml
* Add task_summary.md to docs/es
* Change title of task_summary.md
* Translate firsts paragraphs
* Translate middle paragraphs
* Translte the rest of the doc
* Edit firts paragraph
2024-02-16 15:50:06 -08:00
Matt
2f1003be86
Add chat support to text generation pipeline ( #28945 )
...
* Add chat support to text generation pipeline
* Better handling of single elements
* Deprecate ConversationalPipeline
* stash commit
* Add missing add_special_tokens kwarg
* Update chat templating docs to refer to TextGenerationPipeline instead of ConversationalPipeline
* Add ✨ TF✨ tests
* @require_tf
* Add type hint
* Add specific deprecation version
* Remove unnecessary do_sample
* Remove todo - the discrepancy has been resolved
* Update src/transformers/tokenization_utils_base.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/pipelines/text_generation.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-02-16 16:41:01 +00:00
Lysandre Debut
f497f564bb
Update all references to canonical models ( #29001 )
...
* Script & Manual edition
* Update
2024-02-16 08:16:58 +01:00
Merve Noyan
3f4e79d29c
Mask Generation Task Guide ( #28897 )
...
* Create mask_generation.md
* add h1
* add to toctree
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update mask_generation.md
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
* Update mask_generation.md
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Klaus Hipp <khipp@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Klaus Hipp <khipp@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Klaus Hipp <khipp@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update docs/source/en/tasks/mask_generation.md
* Update mask_generation.md
* Update mask_generation.md
---------
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Co-authored-by: Klaus Hipp <khipp@users.noreply.github.com>
2024-02-14 18:29:49 +00:00
amyeroberts
7252e8d937
[Doc] Fix docbuilder - make BackboneMixin and BackboneConfigMixin importable from utils. ( #29002 )
...
* Trigger doc build
* Test removing references
* Importable from utils
* Trigger another run on a new commit for testing
2024-02-14 10:29:22 +00:00
Andrei Panferov
1ecf5f7c98
AQLM quantizer support ( #28928 )
...
* aqlm init
* calibration and dtypes
* docs
* Readme update
* is_aqlm_available
* Simpler link in docs
* Test TODO real reference
* init _import_structure fix
* AqlmConfig autodoc
* integration aqlm
* integrations in tests
* docstring fix
* legacy typing
* Less typings
* More kernels information
* Performance -> Accuracy
* correct tests
* remoced multi-gpu test
* Update docs/source/en/quantization.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update src/transformers/utils/quantization_config.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Brought back multi-gpu tests
* Update src/transformers/integrations/aqlm.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Update tests/quantization/aqlm_integration/test_aqlm.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
---------
Co-authored-by: Andrei Panferov <blacksamorez@yandex-team.ru>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
2024-02-14 09:25:41 +01:00
NielsRogge
63ffd56d02
Add SiglipForImageClassification and CLIPForImageClassification ( #28952 )
...
* First draft
* Add CLIPForImageClassification
* Remove scripts
* Fix doctests
2024-02-14 08:41:31 +01:00
Jonathan Tow
de6029a059
Add StableLM ( #28810 )
...
* Add `StableLM`
* fix(model): re-create from `huggingface-cli add-new-model-like persimmon`
* fix: re-add changes to address comments
* fix(readme): add links to paper
* fix(tokenization_auto): remove `GPTNeoXTokenizerFastFast` ref
* fix(tests): re-add `@slow` decorator to integration tests
* fix(tests): import slow...
* fix(readme_hd): remove whitespace edit
* fix(tokenizer): auto tokenizer tuple
* skip doctests for `modeling_stablelm`
2024-02-14 07:15:18 +01:00
Klaus Hipp
d90acc1643
[i18n-de] Translate CONTRIBUTING.md to German ( #28954 )
...
* Translate contributing.md to German
* Fix formatting issues in contributing.md
* Address review comments
* Fix capitalization
2024-02-12 13:39:20 -08:00
NielsRogge
78ba9f4617
[Docs] Add video section ( #28958 )
...
Add video section
2024-02-12 19:50:31 +01:00
Klaus Hipp
fe3df9d5b3
[Docs] Add language identifiers to fenced code blocks ( #28955 )
...
Add language identifiers to code blocks
2024-02-12 10:48:31 -08:00
NielsRogge
ef5ab72f4b
[Docs] Update README and default pipelines ( #28864 )
...
* Update README and docs
* Update README
* Update README
2024-02-12 10:21:36 +01:00
Klaus Hipp
2749e479f3
[Docs] Fix broken links and syntax issues ( #28918 )
...
* Fix model documentation links in attention.md
* Fix external link syntax
* Fix target anchor names of section links
* Fix copyright statement comments
* Fix documentation headings
2024-02-08 14:13:35 -08:00
Arthur
115ac94d06
[Core generation] Adds support for static KV cache ( #27931 )
...
Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
2024-02-08 11:50:34 +01:00
Klaus Hipp
33df036917
[Docs] Revert translation of '@slow' decorator ( #28912 )
2024-02-08 03:31:47 +01:00
Klaus Hipp
1c31b7aa3b
[Docs] Add missing language options and fix broken links ( #28852 )
...
* Add missing entries to the language selector
* Add links to the Colab and AWS Studio notebooks for ONNX
* Use anchor links in CONTRIBUTING.md
* Fix broken hyperlinks due to spaces
* Fix links to OpenAI research articles
* Remove confusing footnote symbols from author names, as they are also considered invalid markup
2024-02-06 12:01:01 -08:00
Klaus Hipp
4830f26965
[Docs] Fix backticks in inline code and documentation links ( #28875 )
...
Fix backticks in code blocks and documentation links
2024-02-06 11:15:44 -08:00
nakranivaibhav
2e7c942c81
Adds LlamaForQuestionAnswering class in modeling_llama.py along with AutoModel Support ( #28777 )
...
* This is a test commit
* testing commit
* final commit with some changes
* Removed copy statement
* Fixed formatting issues
* Fixed error added past_key_values in the forward method
* Fixed a trailing whitespace. Damn the formatting rules are strict
* Added the copy statement
2024-02-06 03:41:42 +01:00
amyeroberts
ba3264b4e8
Image Feature Extraction pipeline ( #28216 )
...
* Draft pipeline
* Fixup
* Fix docstrings
* Update doctest
* Update pipeline_model_mapping
* Update docstring
* Update tests
* Update src/transformers/pipelines/image_feature_extraction.py
Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
* Fix docstrings - review comments
* Remove pipeline mapping for composite vision models
* Add to pipeline tests
* Remove for flava (multimodal)
* safe pil import
* Add requirements for pipeline run
* Account for super slow efficientnet
* Review comments
* Fix tests
* Swap order of kwargs
* Use build_pipeline_init_args
* Add back FE pipeline for Vilt
* Include image_processor_kwargs in docstring
* Mark test as flaky
* Update TODO
* Update tests/pipelines/test_pipelines_image_feature_extraction.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Add license header
---------
Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2024-02-05 14:50:07 +00:00
Klaus Hipp
721ee783ca
[Docs] Fix spelling and grammar mistakes ( #28825 )
...
* Fix typos and grammar mistakes in docs and examples
* Fix typos in docstrings and comments
* Fix spelling of `tokenizer` in model tests
* Remove erroneous spaces in decorators
* Remove extra spaces in Markdown link texts
2024-02-02 08:45:00 +01:00
Steven Liu
2418c64a1c
[docs] HfQuantizer ( #28820 )
...
* tidy
* fix path
2024-02-02 08:22:18 +01:00
Steven Liu
abbffc4525
[docs] Backbone ( #28739 )
...
* backbones
* fix path
* fix paths
* fix code snippet
* fix links
2024-02-01 09:16:16 -08:00
Rockerz
23ea6743f2
Add models from deit ( #28302 )
...
* Add modelss
* Add 2 more models
* add models to tocrree
* Add modles
* Update docs/source/ja/model_doc/detr.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/ja/model_doc/deit.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/ja/model_doc/deplot.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* fix bugs
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
2024-02-01 09:15:55 -08:00
Matt
7bc6d76396
Add tip on setting tokenizer attributes ( #28764 )
...
* Add tip on setting tokenizer attributes
* Grammar
* Remove the bit that was causing doc builds to fail
2024-02-01 14:44:58 +00:00
JB (Don)
0d26abdd3a
Adding [T5/MT5/UMT5]ForTokenClassification ( #28443 )
...
* Adding [T5/MT5/UMT5]ForTokenClassification
* Add auto mappings for T5ForTokenClassification and variants
* Adding ForTokenClassification to the list of models
* Adding attention_mask param to the T5ForTokenClassification test
* Remove outdated comment in test
* Adding EncoderOnly and Token Classification tests for MT5 and UMT5
* Fix typo in umt5 string
* Add tests for all the existing MT5 models
* Fix wrong comment in dependency_versions_table
* Reverting change to common test for _keys_to_ignore_on_load_missing
The test is correctly picking up redundant keys in _keys_to_ignore_on_load_missing.
* Removing _keys_to_ignore_on_missing from MT5 since the key is not used in the model
* Add fix-copies to MT5ModelTest
2024-02-01 03:53:49 +01:00
Kian Sierra McGettigan
f7076cd346
Flax mistral ( #26943 )
...
* direct copy from llama work
* mistral modules forward pass working
* flax mistral forward pass with sliding window
* added tests
* added layer collection approach
* Revert "added layer collection approach"
This reverts commit 0e2905bf22fb17b6187a
2024-01-31 14:19:02 +01:00
Matt
415e9a0980
Add tf_keras imports to prepare for Keras 3 ( #28588 )
...
* Port core files + ESM (because ESM code is odd)
* Search-replace in modelling code
* Fix up transfo_xl as well
* Fix other core files + tests (still need to add correct import to tests)
* Fix cookiecutter
* make fixup, fix imports in some more core files
* Auto-add imports to tests
* Cleanup, add imports to sagemaker tests
* Use correct exception for importing tf_keras
* Fixes in modeling_tf_utils
* make fixup
* Correct version parsing code
* Ensure the pipeline tests correctly revert to float32 after each test
* Ensure the pipeline tests correctly revert to float32 after each test
* More tf.keras -> keras
* Add dtype cast
* Better imports of tf_keras
* Add a cast for tf.assign, just in case
* Fix callback imports
2024-01-30 17:26:36 +00:00
Younes Belkada
866253f85e
[HfQuantizer] Move it to "Developper guides" ( #28768 )
...
Update _toctree.yml
2024-01-30 07:20:20 +01:00
Poedator
d78e78a0e4
HfQuantizer class for quantization-related stuff in modeling_utils.py (#26610 )
...
* squashed earlier commits for easier rebase
* rm rebase leftovers
* 4bit save enabled @quantizers
* TMP gptq test use exllama
* fix AwqConfigTest::test_wrong_backend for A100
* quantizers AWQ fixes
* _load_pretrained_model low_cpu_mem_usage branch
* quantizers style
* remove require_low_cpu_mem_usage attr
* rm dtype arg from process_model_before_weight_loading
* rm config_origin from Q-config
* rm inspect from q_config
* fixed docstrings in QuantizationConfigParser
* logger.warning fix
* mv is_loaded_in_4(8)bit to BnbHFQuantizer
* is_accelerate_available error msg fix in quantizer
* split is_model_trainable in bnb quantizer class
* rm llm_int8_skip_modules as separate var in Q
* Q rm todo
* fwd ref to HFQuantizer in type hint
* rm note re optimum.gptq.GPTQQuantizer
* quantization_config in __init__ simplified
* replaced NonImplemented with create_quantized_param
* rm load_in_4/8_bit deprecation warning
* QuantizationConfigParser refactoring
* awq-related minor changes
* awq-related changes
* awq config.modules_to_not_convert
* raise error if no q-method in q-config in args
* minor cleanup
* awq quantizer docstring
* combine common parts in bnb process_model_before_weight_loading
* revert test_gptq
* .process_model_ cleanup
* restore dict config warning
* removed typevars in quantizers.py
* cleanup post-rebase 16 jan
* QuantizationConfigParser classmethod refactor
* rework of handling of unexpected aux elements of bnb weights
* moved q-related stuff from save_pretrained to quantizers
* refactor v1
* more changes
* fix some tests
* remove it from main init
* ooops
* Apply suggestions from code review
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* fix awq issues
* fix
* fix
* fix
* fix
* fix
* fix
* add docs
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update docs/source/en/hf_quantizer.md
* address comments
* fix
* fixup
* Update src/transformers/modeling_utils.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/modeling_utils.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* address final comment
* update
* Update src/transformers/quantizers/base.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/quantizers/auto.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* fix
* add kwargs update
* fixup
* add `optimum_quantizer` attribute
* oops
* rm unneeded file
* fix doctests
---------
Co-authored-by: younesbelkada <younesbelkada@gmail.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2024-01-30 02:48:25 +01:00
Sanchit Gandhi
da3c79b245
[Whisper] Make tokenizer normalization public ( #28136 )
...
* [Whisper] Make tokenizer normalization public
* add to docs
2024-01-29 16:07:35 +00:00
Julien Chaumond
26aa03a252
small doc update for CamemBERT ( #28644 )
2024-01-29 15:46:32 +01:00
