Haylee Schäfer
dbc16f4404
Support loading base64 images in pipelines ( #25633 )
...
* support loading base64 images
* add test
* mention in docs
* remove the logging
* sort imports
* update error message
* Update tests/utils/test_image_utils.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* restructure to catch base64 exception
* doesn't like the newline
* download files
* format
* optimize imports
* guess it needs a space?
* support loading base64 images
* add test
* remove the logging
* sort imports
* restructure to catch base64 exception
* doesn't like the newline
* download files
* optimize imports
* guess it needs a space?
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2023-08-29 19:24:24 +01:00
Sylvain Gugger
68fa9a5937
Skip broken tests
2023-08-24 01:48:53 -04:00
amyeroberts
5e5fa0d88c
Mark flaky tests ( #25463 )
...
Make CI less brittle
2023-08-11 15:26:45 +01:00
amyeroberts
944ddce8bf
Enable passing number of channels when inferring data format ( #25412 )
2023-08-09 17:41:21 +01:00
Matthew Hoffman
d4bd33cc9f
Register ModelOutput subclasses as supported torch.utils._pytree nodes ( #25358 )
...
* Register ModelOutput subclasses as supported torch.utils._pytree nodes
Fixes #25357 where DDP with static_graph=True does not sync gradients when calling backward() over tensors contained in ModelOutput subclasses
* Add test for torch pytree ModelOutput serialization and deserialization
2023-08-08 08:12:11 +02:00
Roland Szabo
d114a6b71f
Add timeout parameter to load_image function ( #25184 )
...
* Add timeout parameter to load_image function.
* Remove line.
* Reformat code
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Add parameter to docs.
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2023-08-03 15:51:54 +01:00
Yih-Dar
1b4f6199c6
Update tiny model info. and pipeline testing ( #25213 )
...
* update tiny_model_summary.json
* update
* update
* update
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-07-31 19:35:33 +02:00
Lucain
c1dba1111b
Add test when downloading from gated repo ( #25039 )
2023-07-28 08:14:27 -04:00
Yih-Dar
3441ad7d43
Make (TF) CI faster (test only a subset of model classes) ( #24592 )
...
* fix
* fix
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-06-30 16:54:54 +02:00
Yih-Dar
e84bf1f734
⚠️ Time to say goodbye to py37 ( #24091 )
...
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-06-28 07:22:39 +02:00
Sylvain Gugger
eb849f6604
Migrate doc files to Markdown. ( #24376 )
...
* Rename index.mdx to index.md
* With saved modifs
* Address review comment
* Treat all files
* .mdx -> .md
* Remove special char
* Update utils/tests_fetcher.py
Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
---------
Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
2023-06-20 18:07:47 -04:00
Yih-Dar
c23d131eab
Update tiny models for pipeline testing. ( #24364 )
...
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-06-20 14:43:10 +02:00
Matt
56efbf4301
TensorFlow CI fixes ( #24360 )
...
* Fix saved_model_creation_extended
* Skip the BLIP model creation test for now
* Fix TF SAM test
* Fix longformer tests
* Fix Wav2Vec2
* Add a skip for XLNet
* make fixup
* make fix-copies
* Add comments
2023-06-20 12:59:21 +01:00
Matt
9138995025
Add test for proper TF input signatures ( #24320 )
...
* Add test for proper input signatures
* No more signature pruning
* Test the dummy inputs are valid too
* fine-tine -> fine-tune
* Fix indent in test_dataset_conversion
2023-06-16 17:03:13 +01:00
Matt
3bd1fe4315
Stop storing references to bound methods via tf.function ( #24146 )
...
* Stop storing references to bound methods in tf.functions
* Remove the gc.collect calls now that we resolved the underlying problem
* Remove the default signature from model.serving entirely, big cleanup
* Remove _prune_signature as self.input_signature can prune itself
* Restore serving docstring
* Update int support test to check the input signature
* Make sure other tests also use model.input_signature and not serving.input_signature
* Restore _prune_signature
* Remove the doctest GC now it's no longer needed
* Correct core tests to use the pruned sig
* order lines correctly in core tests
* Add eager_serving back with a deprecation warning
2023-06-13 19:04:22 +01:00
Radamés Ajna
a73883ae9e
add trust_remote_code option to CLI download cmd ( #24097 )
...
* add trust_remote_code option
* require_torch
2023-06-08 11:13:57 -04:00
Daniel King
89159651ba
Fix the regex in get_imports to support multiline try blocks and excepts with specific exception types ( #23725 )
...
* fix and test get_imports for multiline try blocks, and excepts with specific errors
* fixup
* add some more tests
* add license
2023-05-24 15:40:19 -04:00
Matt
f8b2574416
Better TF docstring types ( #23477 )
...
* Rework TF type hints to use | None instead of Optional[] for tf.Tensor
* Rework TF type hints to use | None instead of Optional[] for tf.Tensor
* Don't forget the imports
* Add the imports to tests too
* make fixup
* Refactor tests that depended on get_type_hints
* Better test refactor
* Fix an old hidden bug in the test_keras_fit input creation code
* Fix for the Deit tests
2023-05-24 13:52:52 +01:00
Yih-Dar
ffad4f1373
Update tiny models and pipeline tests ( #23446 )
...
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-05-18 17:29:04 +02:00
Matthijs Hollemans
7f91950901
audio_utils improvements ( #21998 )
...
* silly change to allow making a PR
* clean up doc comments
* simplify hertz_to_mel and mel_to_hertz
* fixup
* clean up power_to_db
* also add amplitude_to_db
* move functions
* clean up mel_filter_bank
* fixup
* credit librosa & torchaudio authors
* add unit tests
* tests for power_to_db and amplitude_to_db
* add mel_filter_bank tests
* rewrite STFT
* add convenience spectrogram function
* missing transpose
* fewer transposes
* add integration test to M-CTC-T
* frame length can be either window or FFT length
* rewrite stft API
* add preemphasis coefficient
* move argument
* add log option to spectrogram
* replace M-CTC-T feature extractor
* fix api thing
* replace whisper STFT
* replace whisper mel filters
* replace tvlt's stft
* allow alternate window names
* replace speecht5 stft
* fixup
* fix integration tests
* fix doc comments
* remove manual FFT length calculation
* fix docs
* go away, deprecation warnings
* combine everything into spectrogram function
* add deprecated functions back
* fixup
2023-05-09 09:10:17 -04:00
amyeroberts
90e8263d91
Add methods to update and verify out_features out_indices ( #23031 )
...
* Add methods to update and verify out_features out_indices
* Safe update for config attributes
* Fix function names
* Save config correctly
* PR comments - use property setters
* PR comment - directly set attributes
* Update test
* Add updates to recently merged focalnet backbone
2023-05-04 10:15:06 +01:00
Xuehai Pan
ee4bc07474
Support union types X | Y syntax for HfArgumentParser for Python 3.10+ ( #23126 )
...
* Support union types `X | Y` syntax for `HfArgumentParser` for Python 3.10+
* Add tests for PEP 604 for `HfArgumentParser`
* Reorganize tests
2023-05-03 10:49:54 -04:00
Yih-Dar
a4908da04e
Fix the expected error in test_offline_mode_pipeline_exception ( #23022 )
...
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-04-27 14:22:05 +02:00
Lingepumpe
5427250351
Avoid invalid escape sequences, use raw strings ( #22936 )
...
* Avoid invalid escape sequences, use raw strings
* Integrate PR feedback
2023-04-25 09:17:56 -04:00
Lucain
74c55ab9e5
Prepare tests for hfh 0.14 ( #22958 )
...
* Test hf_hub 0.14.0rc1
* fix mocked tests
* package version
---------
Co-authored-by: Sylvain Gugger <Sylvain.gugger@gmail.com>
Co-authored-by: testbot <lucainp@hf.co>
2023-04-24 09:31:50 -04:00
Yih-Dar
975159bb61
Update tiny models and a few fixes ( #22928 )
...
* run_check_tiny_models
* update summary
* update mixin
* update pipeline_model_mapping
* update pipeline_model_mapping
* Update for gpt_bigcode
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-04-24 14:45:22 +02:00
Sylvain Gugger
3876fc6839
Make dynamic code work with offline mode ( #22661 )
...
* Make dynamic code work with offline mode
* Clean up
* Quality
2023-04-10 08:49:42 -04:00
Yih-Dar
c7ec71baf5
Update tiny model summary file for recent models ( #22637 )
...
* Update tiny model summary file for recent models
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-04-06 22:52:59 +02:00
Nicolas Patry
0aa1153ffb
Revert error back into warning for byte fallback conversion. ( #22607 )
2023-04-06 14:00:29 +02:00
Nicolas Patry
1670be4bde
Adding Llama FastTokenizer support. ( #22264 )
...
* Adding Llama FastTokenizer support.
- Requires https://github.com/huggingface/tokenizers/pull/1183 version
- Only support byte_fallback for llama, raise otherwise (safety net).
- Lots of questions are special tokens
How to test:
```python
from transformers.convert_slow_tokenizer import convert_slow_tokenizer
from transformers import AutoTokenizer
from tokenizers import Tokenizer
tokenizer = AutoTokenizer.from_pretrained("huggingface/llama-7b")
if False:
new_tokenizer = Tokenizer.from_file("tok.json")
else:
new_tokenizer = convert_slow_tokenizer(tokenizer)
new_tokenizer.save("tok.json")
strings = [
"This is a test",
"生活的真谛是",
"生活的真谛是[MASK]。",
# XXX: This one is problematic because of special tokens
# "<s> Something something",
]
for string in strings:
encoded = tokenizer(string)["input_ids"]
encoded2 = new_tokenizer.encode(string).ids
assert encoded == encoded2, f"{encoded} != {encoded2}"
decoded = tokenizer.decode(encoded)
decoded2 = new_tokenizer.decode(encoded2)
assert decoded.strip() == decoded2, f"{repr(decoded)} != {repr(decoded2)}"
```
The converter + some test script.
The test script.
Tmp save.
Adding Fast tokenizer + tests.
Adding the tokenization tests.
Correct combination.
Small fix.
Fixing tests.
Fixing with latest update.
Rebased.
fix copies + normalized added tokens + copies.
Adding doc.
TMP.
Doc + split files.
Doc.
Versions + try import.
Fix Camembert + warnings -> Error.
Fix by ArthurZucker.
Not a decorator.
* Fixing comments.
* Adding more to docstring.
* Doc rewriting.
2023-04-06 09:53:03 +02:00
Yih-Dar
8894b81742
Use real tokenizers if tiny version(s) creation has issue(s) ( #22428 )
...
Fix some tiny model creation issues
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-03-29 16:16:23 +02:00
Yih-Dar
e8cc02555e
Automatically create/update tiny models ( #22275 )
...
* Automatically create or update tiny models
* Skip failed tests
* update workflow file
* use revision
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-03-23 19:14:17 +01:00
Yih-Dar
e6de918676
Add Blip and Blip2 for pipeline tests ( #21904 )
...
* fix
* add to tests
* style and quality
* add missing
---------
Co-authored-by: NielsRogge <NielsRogge@users.noreply.github.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-03-02 18:20:34 +01:00
Sylvain Gugger
b29e2dcaff
Fix flaky test for log level ( #21776 )
...
* Fix flaky test for log level
* Fix other flaky test
2023-02-28 16:24:14 -05:00
Yih-Dar
871c31a6f1
🔥 Rework pipeline testing by removing PipelineTestCaseMeta 🚀 ( #21516 )
...
* Add PipelineTesterMixin
* remove class PipelineTestCaseMeta
* move validate_test_components
* Add for ViT
* Add to SPECIAL_MODULE_TO_TEST_MAP
* style and quality
* Add feature-extraction
* update
* raise instead of skip
* add tiny_model_summary.json
* more explicit
* skip tasks not in mapping
* add availability check
* Add Copyright
* A way to diable irrelevant tests
* update with main
* remove disable_irrelevant_tests
* skip tests
* better skip message
* better skip message
* Add all pipeline task tests
* revert
* Import PipelineTesterMixin
* subclass test classes with PipelineTesterMixin
* Add pipieline_model_mapping
* Fix import after adding pipieline_model_mapping
* Fix style and quality after adding pipieline_model_mapping
* Fix one more import after adding pipieline_model_mapping
* Fix style and quality after adding pipieline_model_mapping
* Fix test issues
* Fix import requirements
* Fix mapping for MobileViTModelTest
* Update
* Better skip message
* pipieline_model_mapping could not be None
* Remove some PipelineTesterMixin
* Fix typo
* revert tests_fetcher.py
* update
* rename
* revert
* Remove PipelineTestCaseMeta from ZeroShotAudioClassificationPipelineTests
* style and quality
* test fetcher for all pipeline/model tests
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-02-28 19:40:57 +01:00
Joao Gante
92dfceb124
Inheritance-based framework detection ( #21784 )
2023-02-27 15:31:55 +00:00
Aaron Gokaslan
5e8c8eb5ba
Apply ruff flake8-comprehensions ( #21694 )
2023-02-22 09:14:54 +01:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting ( #21480 )
...
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
2023-02-06 18:10:56 -05:00
Joel Lamy-Poirier
e006ab51ac
Add the GeLU activation from pytorch with the tanh approximation ( #21345 )
...
* gelu_python_tanh
* rename
* Version check, add test
* Pr comment
2023-02-02 09:33:04 -05:00
amyeroberts
d18a1cba24
Accept batched tensor of images as input to image processor ( #21144 )
...
* Accept a batched tensor of images as input
* Add to all image processors
* Update oneformer
2023-01-26 10:15:26 +00:00
Nicolas Patry
25ddd91b24
Fixing offline mode for pipeline (when inferring task). ( #21113 )
...
* Fixing offline mode for pipeline (when inferring task).
* Update src/transformers/pipelines/__init__.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Updating test to reflect change in exception.
* Fixing offline mode.
* Clean.
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2023-01-17 15:24:40 +01:00
amyeroberts
0dde58978a
Rename test_feature_extraction files ( #21140 )
...
* Rename files
* Update file names in tests
2023-01-17 14:04:07 +00:00
Matt
afad0c18d9
Fix TF nightly tests ( #20507 )
...
* Fixed test_saved_model_extended
* Fix TFGPT2 tests
* make fixup
* Make sure keras-nlp utils are available for type hinting too
* Update src/transformers/testing_utils.py
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* make fixup
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
2022-11-30 14:47:54 +00:00
amyeroberts
321ef388fe
Include image processor in add-new-model-like ( #20439 )
2022-11-28 16:46:02 +00:00
Matt
de4159a318
More TF int dtype fixes ( #20384 )
...
* Add a test to ensure int dummy inputs are int64
* Move the test into the existing int64 test and update a lot of existing dummies
* Fix remaining dummies
* Fix remaining dummies
* Test for int64 serving sigs as well
* Update core tests to use tf.int64
* Add better messages to the assertions
* Update all serving sigs to int64
* More sneaky hiding tf.int32s
* Add an optional int32 signature in save_pretrained
* make fixup
* Add Amy's suggestions
* Switch all serving sigs back to tf.int32
* Switch all dummies to tf.int32
* Adjust tests to check for tf.int32 instead of tf.int64
* Fix base dummy_inputs dtype
* Start casting to tf.int32 in input_processing
* Change dtype for unpack_inputs test
* Add proper tf.int32 test
* Make the alternate serving signature int64
2022-11-28 13:24:44 +00:00
Sylvain Gugger
afce73bd9d
Fix ModelOutput instantiation when there is only one tuple ( #20416 )
2022-11-23 15:09:21 -05:00
Konstantin Dobler
1e3f17b5ab
Enhance HfArgumentParser functionality and ease of use ( #20323 )
...
* Enhance HfArgumentParser
* Fix type hints for older python versions
* Fix and add tests (+formatting)
* Add changes
* doc-builder formatting
* Remove unused import "Call"
2022-11-21 12:33:37 -05:00
amyeroberts
a6b7759880
Add Image Processors ( #19796 )
...
* Add CLIP image processor
* Crop size as dict too
* Update warning
* Actually use logger this time
* Normalize doesn't change dtype of input
* Add perceiver image processor
* Tidy up
* Add DPT image processor
* Add Vilt image processor
* Tidy up
* Add poolformer image processor
* Tidy up
* Add LayoutLM v2 and v3 imsge processors
* Tidy up
* Add Flava image processor
* Tidy up
* Add deit image processor
* Tidy up
* Add ConvNext image processor
* Tidy up
* Add levit image processor
* Add segformer image processor
* Add in post processing
* Fix up
* Add ImageGPT image processor
* Fixup
* Add mobilevit image processor
* Tidy up
* Add postprocessing
* Fixup
* Add VideoMAE image processor
* Tidy up
* Add ImageGPT image processor
* Fixup
* Add ViT image processor
* Tidy up
* Add beit image processor
* Add mobilevit image processor
* Tidy up
* Add postprocessing
* Fixup
* Fix up
* Fix flava and remove tree module
* Fix image classification pipeline failing tests
* Update feature extractor in trainer scripts
* Update pad_if_smaller to accept tuple and int size
* Update for image segmentation pipeline
* Update src/transformers/models/perceiver/image_processing_perceiver.py
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
* Update src/transformers/image_processing_utils.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/beit/image_processing_beit.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* PR comments - docstrings; remove accidentally added resize; var names
* Update docstrings
* Add exception if size is not in the right format
* Fix exception check
* Fix up
* Use shortest_edge in tuple in script
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
2022-11-02 11:57:36 +00:00
Sylvain Gugger
d4eb52d13d
Refactor conversion function ( #19799 )
...
* Refactor conversion function
* Remove dupe line
* Fixes
* Fixes
* Use the right variable...
* Fix last test
2022-10-24 13:48:40 -04:00
Sylvain Gugger
fb0bd7b7a8
Fix activations being all the same module ( #19728 )
2022-10-18 11:56:45 -04:00
