Pavel Iakubovskii
48461c0fe2
Make pipeline able to load processor ( #32514 )
...
* Refactor get_test_pipeline
* Fixup
* Fixing tests
* Add processor loading in tests
* Restructure processors loading
* Add processor to the pipeline
* Move model loading on tom of the test
* Update `get_test_pipeline`
* Fixup
* Add class-based flags for loading processors
* Change `is_pipeline_test_to_skip` signature
* Skip t5 failing test for slow tokenizer
* Fixup
* Fix copies for T5
* Fix typo
* Add try/except for tokenizer loading (kosmos-2 case)
* Fixup
* Llama not fails for long generation
* Revert processor pass in text-generation test
* Fix docs
* Switch back to json file for image processors and feature extractors
* Add processor type check
* Remove except for tokenizers
* Fix docstring
* Fix empty lists for tests
* Fixup
* Fix load check
* Ensure we have non-empty test cases
* Update src/transformers/pipelines/__init__.py
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Update src/transformers/pipelines/base.py
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Rework comment
* Better docs, add note about pipeline components
* Change warning to error raise
* Fixup
* Refine pipeline docs
---------
Co-authored-by: Lysandre Debut <hi@lysand.re>
2024-10-09 16:46:11 +01:00
amyeroberts
1de7dc7403
Skip tests properly ( #31308 )
...
* Skip tests properly
* [test_all]
* Add 'reason' as kwarg for skipTest
* [test_all] Fix up
* [test_all]
2024-06-26 21:59:08 +01:00
Matt
065729a692
Remove ConversationalPipeline and Conversation object ( #31165 )
...
* Remove ConversationalPipeline and Conversation object, as they have been deprecated for some time and are due for removal
* Update not-doctested.txt
* Fix JA and ZH docs
* Fix JA and ZH docs some more
* Fix JA and ZH docs some more
2024-06-07 17:50:18 +01:00
Marc Sun
8366b57241
Fix accelerate failing tests ( #30836 )
...
* Fix accelerate tests
* fix clip
* skip dbrx tests
* fix GPTSan
* fix M2M100Model
* same fix as jamba
* fix mt5
* Fix T5Model
* Fix umt5 model
* fix switch_transformers
* fix whisper
* fix gptsan again
* fix siglip recent test
* skip siglip tests
* wrong place fixed
2024-05-23 17:18:58 +02:00
Lysandre Debut
39114c0383
Remove static pretrained maps from the library's internals ( #29112 )
...
* [test_all] Remove static pretrained maps from the library's internals
* Deprecate archive maps instead of removing them
* Revert init changes
* [test_all] Deprecate instead of removing
* [test_all] PVT v2 support
* [test_all] Tests should all pass
* [test_all] Style
* Address review comments
* Update src/transformers/models/deprecated/_archive_maps.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/models/deprecated/_archive_maps.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* [test_all] trigger tests
* [test_all] LLAVA
* [test_all] Bad rebase
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2024-03-25 10:33:38 +01:00
Lysandre Debut
f497f564bb
Update all references to canonical models ( #29001 )
...
* Script & Manual edition
* Update
2024-02-16 08:16:58 +01:00
JB (Don)
0d26abdd3a
Adding [T5/MT5/UMT5]ForTokenClassification ( #28443 )
...
* Adding [T5/MT5/UMT5]ForTokenClassification
* Add auto mappings for T5ForTokenClassification and variants
* Adding ForTokenClassification to the list of models
* Adding attention_mask param to the T5ForTokenClassification test
* Remove outdated comment in test
* Adding EncoderOnly and Token Classification tests for MT5 and UMT5
* Fix typo in umt5 string
* Add tests for all the existing MT5 models
* Fix wrong comment in dependency_versions_table
* Reverting change to common test for _keys_to_ignore_on_load_missing
The test is correctly picking up redundant keys in _keys_to_ignore_on_load_missing.
* Removing _keys_to_ignore_on_missing from MT5 since the key is not used in the model
* Add fix-copies to MT5ModelTest
2024-02-01 03:53:49 +01:00
jiqing-feng
1d7f406e19
fix assisted decoding assistant model inputs ( #27503 )
...
* fix assisted decoding attention_cat
* fix attention_mask for assisted decoding
* fix attention_mask len
* fix attn len
* Use a more clean way to prepare assistant models inputs
* fix param meaning
* fix param name
* fix assistant model inputs
* update token type ids
* fix assistant kwargs copy
* add encoder-decoder tests of assisted decoding
* check if assistant kwargs contains updated keys
* revert test
* fix whisper tests
* fix assistant kwargs
* revert whisper test
* delete _extend funcs
2023-11-27 14:23:54 +00:00
Arthur
186c077513
[CI-test_torch] skip test_tf_from_pt_safetensors and test_assisted_decoding_sample ( #27508 )
...
* skip 4 tests
* nits
* style
* wow it's not my day
* skip new failing tests
* style
* skip for NLLB MoE as well
2023-11-15 08:39:29 +01:00
Matt
14b04b4b9c
Conversation pipeline fixes ( #26795 )
...
* Adjust length limits and allow naked conversation list inputs
* Adjust length limits and allow naked conversation list inputs
* Maybe use a slightly more reasonable limit than 1024
* Skip tests for old models that never supported this anyway
* Cleanup input docstrings
* More docstring cleanup + skip failing TF test
* Make fixup
2023-10-16 17:27:45 +01:00
fxmarty
da971b2271
Keep relevant weights in fp32 when model._keep_in_fp32_modules is set even when accelerate is not installed ( #26225 )
...
* fix bug where weight would not be kept in fp32
* nit
* address review comments
* fix test
2023-09-21 19:00:03 +09:00
Yih-Dar
bd90cda9a6
CI with num_hidden_layers=2 🚀 🚀 🚀 ( #25266 )
...
* CI with layers=2
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-02 20:22:36 +02:00
Yih-Dar
1b4f6199c6
Update tiny model info. and pipeline testing ( #25213 )
...
* update tiny_model_summary.json
* update
* update
* update
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-07-31 19:35:33 +02:00
Sebastian Husch Lee
8f36ab3e22
[T5, MT5, UMT5] Add [T5, MT5, UMT5]ForSequenceClassification ( #24726 )
...
* Initial addition of t5forsequenceclassification
* Adding imports and adding tests
* Formatting
* Running make fix-copies
* Adding mt5forseq
* Formatting
* run make fix-copies
* Adding to docs
* Add model_parallel
* Fix bug
* Fix
* Remove TODO
* Fixing tests for T5ForSequenceClassification
* Undo changes to dependency_versions_table.py
* Change classification head to work with T5Config directly
* Change seq length to let tests pass
* PR comments for formatting
* Formatting
* Initial addition of UMT5ForSequenceClassification
* Adding to inits and formatting
* run make fix-copies
* Add doc for UMT5ForSeqClass
* Update UMT5 config
* Fix docs
* Skip torch fx test for SequenceClassification
* Formatting
* Add skip to UMT5 tests as well
* Fix umt5 tests
* Running make fix-copies
* PR comments
* Fix for change to sentence_representation
* Rename seq_len to hidden_size since that's what it is
* Use base_model to follow format of the rest of the library
* Update docs
* Extract the decoder_input_ids changes and make one liner
* Make one-liner
2023-07-25 21:02:49 +02:00
Sebastian
06910f5a76
[T5] Add T5ForQuestionAnswering and MT5ForQuestionAnswering ( #24481 )
...
* Adding T5ForQuestionAnswering
* Changed weight initialization that results in better initial loss when fine-tuning
* Update to class variables
* Running make fixup
* Running make fix-copies
* Remove model_parallel
* Adding MT5ForQuestionAnswering
* Adding docs
* Fix wrong doc
* Update src/transformers/models/mt5/modeling_mt5.py
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update src/transformers/models/t5/modeling_t5.py
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* File formatting
* Undoing change
---------
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
2023-06-27 10:07:06 -04:00
Yih-Dar
fa01127a67
update_pip_test_mapping ( #22606 )
...
* Add TFBlipForConditionalGeneration
* update pipeline_model_mapping
* Add import
* Revert changes in GPTSanJapaneseTest
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-04-06 17:56:06 +02:00
Yih-Dar
871c31a6f1
🔥 Rework pipeline testing by removing PipelineTestCaseMeta 🚀 ( #21516 )
...
* Add PipelineTesterMixin
* remove class PipelineTestCaseMeta
* move validate_test_components
* Add for ViT
* Add to SPECIAL_MODULE_TO_TEST_MAP
* style and quality
* Add feature-extraction
* update
* raise instead of skip
* add tiny_model_summary.json
* more explicit
* skip tasks not in mapping
* add availability check
* Add Copyright
* A way to diable irrelevant tests
* update with main
* remove disable_irrelevant_tests
* skip tests
* better skip message
* better skip message
* Add all pipeline task tests
* revert
* Import PipelineTesterMixin
* subclass test classes with PipelineTesterMixin
* Add pipieline_model_mapping
* Fix import after adding pipieline_model_mapping
* Fix style and quality after adding pipieline_model_mapping
* Fix one more import after adding pipieline_model_mapping
* Fix style and quality after adding pipieline_model_mapping
* Fix test issues
* Fix import requirements
* Fix mapping for MobileViTModelTest
* Update
* Better skip message
* pipieline_model_mapping could not be None
* Remove some PipelineTesterMixin
* Fix typo
* revert tests_fetcher.py
* update
* rename
* revert
* Remove PipelineTestCaseMeta from ZeroShotAudioClassificationPipelineTests
* style and quality
* test fetcher for all pipeline/model tests
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-02-28 19:40:57 +01:00
Younes Belkada
ae9230af40
[T5] Fix torchquant issue ( #21843 )
...
* fix torchquant issue
* add tests
2023-02-28 15:09:44 +01:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting ( #21480 )
...
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
2023-02-06 18:10:56 -05:00
Younes Belkada
1af4bee896
Add keep_in_fp32_modules support ( #20683 )
...
* add `keep_in_fp32_modules` support
* pass it as class attribute
* few modifs
- make tests `slow`
- fix logic
* better logic
* fix failing test
* `bfloat16` support
* Update src/transformers/modeling_utils.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* fix
* simplify tests
* simplify tests
* fix test
* modify message
* more checks
* fix failing tests
* add more conditions
- add `is_accelerate_available`
- fixes pipleine tests that failed
* add suggestions
* Update src/transformers/modeling_utils.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* fix failing `bnb` test
* add last safety checker
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2022-12-13 11:59:57 +01:00
Joao Gante
f270b960d6
Generate: move generation_*.py src files into generation/*.py ( #20096 )
...
* move generation_*.py src files into generation/*.py
* populate generation.__init__ with lazy loading
* move imports and references from generation.xxx.object to generation.object
2022-11-09 15:34:08 +00:00
Joao Gante
831590f6a9
Generate: contrastive search with full optional outputs ( #19963 )
...
* Use beam search functionality; Add extra outputs and test
* Add full tests for contrastive search
* Add error message on unconventional cache format
2022-11-01 18:15:36 +00:00
Sylvain Gugger
9e72eb4416
Skip disk offload test for T5
2022-06-07 11:11:40 -04:00
Sylvain Gugger
8343901263
Fix all offload and MP tests ( #17533 )
2022-06-03 09:59:13 -04:00
DanielHesslow
607acd4fbd
Add Gated-SiLU to T5 ( #17420 )
...
* Add gated-silu to t5 architecture to support UL2
* Fix error message
* formatting
* formatting again
* refactor
* fix classnames in _init_weights
* remove is_gated
* add test
* fix test
* Try without the test?
* Add back the test.
* Improve error message.
Co-authored-by: Daniel Hesslow <daniel@lighton.ai>
2022-06-03 10:56:37 +02:00
Michael Benayoun
28d0048218
Fx support for multiple model architectures ( #17393 )
...
* Support for Bart and LayoutLM, and partial support for XLNet
* Support for mbart
* A lot of new models supported
* Support for other models
* LayoutLM fix
* Use strings instead of classes
2022-05-31 10:02:55 +02:00
Sylvain Gugger
afe5d42d8d
Black preview ( #17217 )
...
* Black preview
* Fixup too!
* Fix check copies
* Use the same version as the CI
* Bump black
2022-05-12 16:25:55 -04:00
Yih-Dar
19420fd99e
Move test model folders ( #17034 )
...
* move test model folders (TODO: fix imports and others)
* fix (potentially partially) imports (in model test modules)
* fix (potentially partially) imports (in tokenization test modules)
* fix (potentially partially) imports (in feature extraction test modules)
* fix import utils.test_modeling_tf_core
* fix path ../fixtures/
* fix imports about generation.test_generation_flax_utils
* fix more imports
* fix fixture path
* fix get_test_dir
* update module_to_test_file
* fix get_tests_dir from wrong transformers.utils
* update config.yml (CircleCI)
* fix style
* remove missing imports
* update new model script
* update check_repo
* update SPECIAL_MODULE_TO_TEST_MAP
* fix style
* add __init__
* update self-scheduled
* fix add_new_model scripts
* check one way to get location back
* python setup.py build install
* fix import in test auto
* update self-scheduled.yml
* update slack notification script
* Add comments about artifact names
* fix for yolos
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2022-05-03 14:42:02 +02:00
