riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity
14,237 Commits 887 Branches 232 Tags 2.1 GiB
eec5a3a8d8
Commit Graph

14237 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rockerz
eec5a3a8d8
Refactor code part in documentation translated to japanese (#26900) 
Refactor code in documentation
 2023-10-18 10:35:58 -07:00
Matt
d933818d67
Add default template warning (#26637) 
* Add default template warnings

* make fixup

* Move warnings to FutureWarning

* Move warnings to FutureWarning

* fix make fixup

* Remove futurewarning
 2023-10-18 17:38:52 +01:00
Matt
de55ead1f1
Emergency PR to skip conversational tests to fix CI (#26906) 2023-10-18 15:33:43 +01:00
Arthur
ef7e93699a
[Tokenizer] Fix slow and fast serialization (#26570) 
* fix

* last attempt

* current work

* fix forward compatibility

* save all special tokens

* current state

* revert additional changes

* updates

* remove tokenizer.model

* add a test and the fix

* nit

* revert one more break

* fix typefield issue

* quality

* more tests

* fix fields for FC

* more nits?

* new additional changes

* how

* some updates

* simplify all

* more nits

* revert some things to original

* nice

* nits

* a small hack

* more nits

* ahhaha

* fixup

* update

* make test run on ci

* use subtesting

* update

* Update .circleci/create_circleci_config.py

* updates

* fixup

* nits

* replace typo

* fix the test

* nits

* update

* None max dif pls

* a partial fix

* had to revert one thing

* test the fast

* updates

* fixup

* and more nits

* more fixes

* update

* Oupsy 👁️

* nits

* fix marian

* on our way to heaven

* Update src/transformers/models/t5/tokenization_t5.py

Co-authored-by: Lysandre Debut <hi@lysand.re>

* fixup

* Update src/transformers/tokenization_utils_fast.py

Co-authored-by: Leo Tronchon <leo.tronchon@gmail.com>

* Update src/transformers/tokenization_utils_base.py

Co-authored-by: Leo Tronchon <leo.tronchon@gmail.com>

* fix phobert

* skip some things, test more

* nits

* fixup

* fix deberta

* update

* update

* more updates

* skip one test

* more updates

* fix camembert

* can't test this one

* more good fixes

* kind of a major update

- seperate what is only done in fast in fast init and refactor
- add_token(AddedToken(..., speicla = True)) ignores it in fast
- better loading

* fixup

* more fixups

* fix pegasus and mpnet

* remove skipped tests

* fix phoneme tokenizer if self.verbose

* fix individual models

* update common tests

* update testing files

* all over again

* nits

* skip test for markup lm

* fixups

* fix order of addition in fast by sorting the added tokens decoder

* proper defaults for deberta

* correct default for fnet

* nits on add tokens, string initialized to special if special

* skip irrelevant herbert tests

* main fixes

* update test added_tokens_serialization

* the fix for bart like models and class instanciating

* update bart

* nit!

* update idefix test

* fix whisper!

* some fixup

* fixups

* revert some of the wrong chanegs

* fixup

* fixup

* skip marian

* skip the correct tests

* skip for tf and flax as well

---------

Co-authored-by: Lysandre Debut <hi@lysand.re>
Co-authored-by: Leo Tronchon <leo.tronchon@gmail.com>
 2023-10-18 16:30:53 +02:00
Matt
34678db4a1
Fix Seq2seqTrainer decoder attention mask (#26841) 
Don't drop decoder_input_ids without also dropping decoder_attention_mask
 2023-10-18 13:28:15 +01:00
Merve Noyan
280c757f6c
Knowledge distillation for vision guide (#25619) 
* Knowledge distillation for vision guide

* Update knowledge_distillation_for_image_classification.md

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: Rafael Padilla <31217453+rafaelpadilla@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: Rafael Padilla <31217453+rafaelpadilla@users.noreply.github.com>

* Iterated on Rafael's comments

* Added to toctree

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: Rafael Padilla <31217453+rafaelpadilla@users.noreply.github.com>

* Addressed comments

* Update knowledge_distillation_for_image_classification.md

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: Rafael Padilla <31217453+rafaelpadilla@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update knowledge_distillation_for_image_classification.md

* Update knowledge_distillation_for_image_classification.md

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* Update docs/source/en/tasks/knowledge_distillation_for_image_classification.md

Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* Address comments

* Update knowledge_distillation_for_image_classification.md

* Explain KL Div

---------

Co-authored-by: Rafael Padilla <31217453+rafaelpadilla@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
 2023-10-18 04:42:32 -07:00
dependabot[bot]
bece55d8f9
Bump urllib3 from 1.26.17 to 1.26.18 in /examples/research_projects/decision_transformer (#26889) 
Bump urllib3 in /examples/research_projects/decision_transformer

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.17...1.26.18)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-18 13:31:06 +02:00
dependabot[bot]
6d644d6852
Bump urllib3 from 1.26.17 to 1.26.18 in /examples/research_projects/visual_bert (#26890) 
Bump urllib3 in /examples/research_projects/visual_bert

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.17...1.26.18)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-18 04:30:50 -07:00
Joao Gante
e893b1efbb
Generate: improve docstrings for custom stopping criteria (#26863) 
improve docstrings
 2023-10-18 09:55:01 +01:00
jayfurmanek
ef42cb6274
Fix TensorFlow pakage check (#26842) 
Add tf-nightly-rocm to _is_tf_available check
 2023-10-17 23:15:50 +01:00
Rockerz
b002353dca
Translating en/internal folder docs to Japanese 🇯🇵 (#26747) 
* Add translation to fitst 3 file of internal folder

* Update Toctree.md and add files

* Update docs/source/ja/internal/generation_utils

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Rename generation_utils file

* rename pipelines_utils.md

* Change file names

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-10-17 15:01:21 -07:00
Bingchen Zhao
46092f763d
Fixed a typo in mistral.md (#26879) 
Fix a typo in mistral.md
 2023-10-17 14:06:37 -07:00
louietouie
51042ae8e5
[docstring] Fix docstring for LukeConfig (#26858) 
* Deleted LukeConfig and ran check_docstrings.py

* Filled docstring information

---------

Co-authored-by: louie <louisparizeau@Chicken.local>
 2023-10-17 19:30:46 +02:00
Yoach Lacombe
db611aabee
🚨 🚨 Raise error when no speaker embeddings in speecht5._generate_speech (#26418) 
* add warning when no speaker embeddings in speecht5._generate_speech

* modify warning to error

* adapt generation test
 2023-10-17 15:59:35 +02:00
Younes Belkada
41c42f85f6
[FA2] Fix flash attention 2 fine-tuning with Falcon (#26852) 
fix fa2 + dropout issue
 2023-10-17 15:38:03 +02:00
Joao Gante
4b423e6074
🚨🚨 Generate: change order of ops in beam sample to avoid nans (#26843) 
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
 2023-10-17 10:32:49 +01:00
larekrow
0b8604d002
Update logits_process.py docstrings to clarify penalty and reward cases (attempt #2) (#26784) 
* Update logits_process.py docstrings + match arg fields to __init__'s

* Ran `make style`
 2023-10-17 10:13:37 +02:00
Shinji Yamada
85e9d64480
fix: when window_size is passes as array (#26800) 2023-10-17 09:26:03 +02:00
Susheel Thapa
b3961f7291
Chore: Typo fixed in multiple files of docs/source/en/model_doc (#26833) 
* Chore: Typo fixed in multiple files of docs/source/en/model_doc

* Update docs/source/en/model_doc/nllb-moe.md

Co-authored-by: Aryan V S <avs050602@gmail.com>

---------

Co-authored-by: Aryan V S <avs050602@gmail.com>
 2023-10-17 07:10:08 +02:00
Yih-Dar
b8f1cde931
Fix Mistral OOM again (#26847) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-16 22:47:20 +02:00
Younes Belkada
fd6a0ade9b
🚨🚨🚨 [Quantization] Store the original dtype in the config as a private attribute 🚨🚨🚨 (#26761) 
* First step

* fix

* add adjustements for gptq

* change to `_pre_quantization_dtype`

* Update src/transformers/modeling_utils.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* fix serialization

* Apply suggestions from code review

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* fixup

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
 2023-10-16 19:56:53 +02:00
Matt
14b04b4b9c
Conversation pipeline fixes (#26795) 
* Adjust length limits and allow naked conversation list inputs

* Adjust length limits and allow naked conversation list inputs

* Maybe use a slightly more reasonable limit than 1024

* Skip tests for old models that never supported this anyway

* Cleanup input docstrings

* More docstring cleanup + skip failing TF test

* Make fixup
 2023-10-16 17:27:45 +01:00
przemL
5c6b83cb69
[docstring] Fix bert generation tokenizer (#26820) 
* Remove BertGenerationTokenizer from objects to ignore

The file BertGenerationTokenizer is removed from
objects to ignore as a first step to fix docstring.

* Docstrings fix for BertGenerationTokenizer

Docstring fix is generated for BertGenerationTokenizer
by using check_docstrings.py.

* Fix docstring for BertGenerationTokenizer

Added sep_token type and docstring in BertGenerationTokenizer.
 2023-10-16 18:26:55 +02:00
Yih-Dar
12cc123359
Better way to run AMD CI with different flavors (#26634) 
* Enable testing against mi250

* Change BERT to trigger tests

* Revert BERT's change

* AMD CI

* AMD CI

---------

Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-16 16:24:30 +02:00
Pedro Cuenca
3ef7134553
Llama tokenizer: remove space in template comment (#26788) 
* Remove space in template comment

I think the space between the eos and bos tokens is not present in the actual template output. I'm using this documentation as a reference for everyone asking about prompting, so would like to clarify whether there's a space or not :)

* Update fast tokenizer too

* Apply to Code Llama

* Link to original code snippet.
 2023-10-16 15:16:03 +01:00
Patrick von Platen
805d5d2111
Add LLM doc (#26058) 
* [WIP] Add LLM doc

* rename

* latex

* latex

* Fix more latex

* [LLMs] Getting most out of LLMS

* improve

* try again

* Apply suggestions from code review

Co-authored-by: Maria Khalusova <kafooster@gmail.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* Update docs/source/en/llm_tutorial_optimization.md

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>

* Apply suggestions from code review

* move file

---------

Co-authored-by: Maria Khalusova <kafooster@gmail.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
 2023-10-16 16:09:50 +02:00
NielsRogge
570b3f9cdd
[OWL-ViT, OWLv2] Add resources (#26822) 
Add resources
 2023-10-16 15:47:44 +02:00
Jintao
b91cff5a3e
fix resume_from_checkpoint bug (#26739) 
* fix resume_from_checkpoint bug

* update code
 2023-10-16 15:29:47 +02:00
Sourab Mangrulkar
a5f5568d75
Make fsdp ram efficient loading optional (#26631) 
make fsdp ram efficient loading optional
 2023-10-16 06:29:01 -07:00
Merve Noyan
5d997f227c
Image-to-Image Task Guide (#26595) 
* img2img task guide

* Update year

* Add to toctree

* Update docs/source/en/tasks/image_to_image.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/image_to_image.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/image_to_image.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/image_to_image.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Update docs/source/en/tasks/image_to_image.md

Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* Update docs/source/en/tasks/image_to_image.md

Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* Addressed comments

* Update docs/source/en/tasks/image_to_image.md

Co-authored-by: Maria Khalusova <kafooster@gmail.com>

* Addressed comments

---------

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Maria Khalusova <kafooster@gmail.com>
 2023-10-16 15:12:03 +02:00
Bojun-Feng
5c081e2993
[docstring] Fix docstring for CodeLlamaTokenizerFast (#26666) 
* remove from OBJECTS_TO_IGNORE

* run check_docstrings.py

* fill in information

* ignore CodeLlamaTokenizer
 2023-10-16 10:11:45 +02:00
Yuki Shindo
69a26c7ecd
Add Japanese translation (#26799) 
Translated into Japanese (README_ja)
 2023-10-16 10:10:23 +02:00
Sparty
0e52af4d7b
[docstring] Fix docstring for CanineConfig (#26771) 
* Remove CanineConfig from check_docstrings

* Run fix_and_overwrite for CanineConfig

* Replace <fill_type> and <fill_docstring> in configuration_canine.py with type and docstring values

---------

Co-authored-by: vignesh-raghunathan <vignesh_raghunathan@intuit.com>
 2023-10-16 10:08:44 +02:00
Shreyas S
0dd58d96a0
Fixed typos (#26810) 
Update feature_extractor.md
 2023-10-16 09:52:29 +02:00
KRØNE
21dc585942
translation brazilian portuguese (#26769) 
* add translation brazilian portuguese

* add translation brazilian portuguese

* add translation brazilian portuguese title

* add translation portuguese tag

* Update README_pt-br.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update README_pt-br.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update README_pt-br.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update README_pt-br.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-10-13 11:13:47 -07:00
Injin Paek
d6e5b02ef3
Add CLIP resources (#26534) 
* docs: feat: model resources for CLIP

* fix: resolve suggestion

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fix: resolve suggestion

* fix: resolve suggestion

* fix: resolve suggestion

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fix: resolve suggestion

* fix: resolve suggestions

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-10-13 11:12:59 -07:00
Younes Belkada
7cc6f822a3
[Flava] Fix flava doc (#26789) 
* fix flava doctest

* add shape

* adapt
 2023-10-13 18:38:36 +02:00
Matteo Raso
8e05ad326b
Fixed KeyError for Mistral (#26682) 
* Fixed KeyError for Mistral

* Removed try block

* Removed whitespace
 2023-10-13 17:20:26 +02:00
NielsRogge
762af3e3c7
Add OWLv2, bis (#26668) 
* First draft

* Update conversion script

* Update copied from statements

* Fix style

* Add copied from to config

* Add copied from to processor

* Run make fixup

* Add docstring

* Update docstrings

* Add method

* Improve docstrings

* Fix docstrings

* Improve docstrings

* Remove onnx

* Add flag

* Address comments

* Add copied from to model tests

* Add flag to conversion script

* Add code snippet

* Address more comments

* Address comment

* Improve conversion script

* More improvements

* Add expected objectness logits

* Skip test

* Improve conversion script

* Extend conversion script

* Convert large checkpoint

* Fix doc tests

* Convert all checkpoints, update integration tests

* Add checkpoint_path arg

* Fix repo_id
 2023-10-13 16:41:24 +02:00
Matt
bdb391e9c6
Fix Falcon generation test (#26770) 2023-10-13 15:10:27 +01:00
Matt
c9785d956b
Disable default system prompt for LLaMA (#26765) 
* Disable default system prompt for LLaMA

* Update test to not expect default prompt
 2023-10-13 14:48:38 +01:00
Younes Belkada
6df9179c1c
[core] Fix fa-2 import (#26785) 
* fix fa-2 import

* nit
 2023-10-13 12:56:50 +02:00
dekomori_sanae09
5bfda28dd3
[docstring] fix docstring DPRConfig (#26674) 
* fix docstring dpr config

* fix style

* Update descp

Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>

---------

Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
 2023-10-13 12:13:43 +02:00
Yih-Dar
288bf5c1d2
Fix num. of minimal calls to the Hub with peft for pipeline (#26385) 
* fix

* [skip-ci] fix

* [skip-ci] fix

* [skip-ci] fix

* [skip-ci] fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-13 11:03:14 +02:00
Bojun-Feng
d085662c59
[docstring] Fix docstring for RwkvConfig (#26782) 
* update check_docstrings

* update docstring
 2023-10-13 10:20:30 +02:00
Yih-Dar
21da3b2461
Update expect outputs of IdeficsProcessorTest.test_tokenizer_padding (#26779) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-13 09:52:10 +02:00
Wonhyeong Seo
7790943c91
🌐 [i18n-KO] Translated big_models.md to Korean (#26245) 
* docs: ko: big_models.md

* feat: chatgpt draft

* fix: manual edits

* fix: resolve suggestions

Co-Authored-By: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-Authored-By: heuristicwave <31366038+heuristicwave@users.noreply.github.com>
Co-Authored-By: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-Authored-By: heuristicwave <31366038+heuristicwave@users.noreply.github.com>
Co-Authored-By: bolizabeth <68984363+bolizabeth@users.noreply.github.com>

---------

Co-authored-by: bolizabeth <68984363+bolizabeth@users.noreply.github.com>
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-authored-by: heuristicwave <31366038+heuristicwave@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-10-12 15:00:12 -07:00
Yih-Dar
3e93dd295b
Skip TrainerIntegrationFSDP::test_basic_run_with_cpu_offload if torch < 2.1 (#26764) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-12 18:22:09 +02:00
Heinz-Alexander Fuetterer
883ed4b344
chore: fix typos (#26756) 2023-10-12 18:00:27 +02:00
Yih-Dar
a243cdca2a
Fix PerceiverModelIntegrationTest::test_inference_masked_lm (#26760) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-12 17:43:06 +02:00