* add distribution head to forecasting
* formatting
* Add generate function for forecasting
* Add generate function to prediction task
* formatting
* use argsort
* add past_observed_mask ordering
* fix arguments
* docs
* add back test_model_outputs_equivalence test
* formatting
* cleanup
* formatting
* use ACT2CLS
* formatting
* fix add_start_docstrings decorator
* add distribution head and generate function to regression task
add distribution head and generate function to regression task. Also made add PatchTSTForForecastingOutput, PatchTSTForRegressionOutput.
* add distribution head and generate function to regression task
add distribution head and generate function to regression task. Also made add PatchTSTForForecastingOutput, PatchTSTForRegressionOutput.
* fix typos
* add forecast_masking
* fixed tests
* use set_seed
* fix doc test
* formatting
* Update docs/source/en/model_doc/patchtst.md
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* better var names
* rename PatchTSTTranspose
* fix argument names and docs string
* remove compute_num_patches and unused class
* remove assert
* renamed to PatchTSTMasking
* use num_labels for classification
* use num_labels
* use default num_labels from super class
* move model_type after docstring
* renamed PatchTSTForMaskPretraining
* bs -> batch_size
* more review fixes
* use hidden_state
* rename encoder layer and block class
* remove commented seed_number
* edit docstring
* Add docstring
* formatting
* use past_observed_mask
* doc suggestion
* make fix-copies
* use Args:
* add docstring
* add docstring
* change some variable names and add PatchTST before some class names
* formatting
* fix argument types
* fix tests
* change x variable to patch_input
* format
* formatting
* fix-copies
* Update tests/models/patchtst/test_modeling_patchtst.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* move loss to forward
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* formatting
* fix a bug when pre_norm is set to True
* output_hidden_states is set to False as default
* set pre_norm=True as default
* format docstring
* format
* output_hidden_states is None by default
* add missing docs
* better var names
* docstring: remove default to False in output_hidden_states
* change labels name to target_values in regression task
* format
* fix tests
* change to forecast_mask_ratios and random_mask_ratio
* change mask names
* change future_values to target_values param in the prediction class
* remove nn.Sequential and make PatchTSTBatchNorm class
* black
* fix argument name for prediction
* add output_attentions option
* add output_attentions to PatchTSTEncoder
* formatting
* Add attention output option to all classes
* Remove PatchTSTEncoderBlock
* create PatchTSTEmbedding class
* use config in PatchTSTPatchify
* Use config in PatchTSTMasking class
* add channel_attn_weights
* Add PatchTSTScaler class
* add output_attentions arg to test function
* format
* Update doc with image patchtst.md
* fix-copies
* rename Forecast <-> Prediction
* change name of a few parameters to match with PatchTSMixer.
* Remove *ForForecasting class to match with other time series models.
* make style
* Remove PatchTSTForForecasting in the test
* remove PatchTSTForForecastingOutput class
* change test_forecast_head to test_prediction_head
* style
* fix docs
* fix tests
* change num_labels to num_targets
* Remove PatchTSTTranspose
* remove arguments in PatchTSTMeanScaler
* remove arguments in PatchTSTStdScaler
* add config as an argument to all the scaler classes
* reformat
* Add norm_eps for batchnorm and layernorm
* reformat.
* reformat
* edit docstring
* update docstring
* change variable name pooling to pooling_type
* fix output_hidden_states as tuple
* fix bug when calling PatchTSTBatchNorm
* change stride to patch_stride
* create PatchTSTPositionalEncoding class and restructure the PatchTSTEncoder
* formatting
* initialize scalers with configs
* edit output_hidden_states
* style
* fix forecast_mask_patches doc string
* doc improvements
* move summary to the start
* typo
* fix docstring
* turn off masking when using prediction, regression, classification
* return scaled output
* adjust output when using distribution head
* remove _num_patches function in the config
* get config.num_patches from patchifier init
* add output_attentions docstring, remove tuple in output_hidden_states
* change SamplePatchTSTPredictionOutput and SamplePatchTSTRegressionOutput to SamplePatchTSTOutput
* remove print("model_class: ", model_class)
* change encoder_attention_heads to num_attention_heads
* change norm to norm_layer
* change encoder_layers to num_hidden_layers
* change shared_embedding to share_embedding, shared_projection to share_projection
* add output_attentions
* more robust check of norm_type
* change dropout_path to path_dropout
* edit docstring
* remove positional_encoding function and add _init_pe in PatchTSTPositionalEncoding
* edit shape of cls_token and initialize it
* add a check on the num_input_channels.
* edit head_dim in the Prediction class to allow the use of cls_token
* remove some positional_encoding_type options, remove learn_pe arg, initalize pe
* change Exception to ValueError
* format
* norm_type is "batchnorm"
* make style
* change cls_token shape
* Change forecast_mask_patches to num_mask_patches. Remove forecast_mask_ratios.
* Bring PatchTSTClassificationHead on top of PatchTSTForClassification
* change encoder_ffn_dim to ffn_dim and edit the docstring.
* update variable names to match with the config
* add generation tests
* change num_mask_patches to num_forecast_mask_patches
* Add examples explaining the use of these models
* make style
* Revert "Revert "[time series] Add PatchTST (#25927)" (#27486)"
This reverts commit 78f6ed6c70.
* make style
* fix default std scaler's minimum_scale
* fix docstring
* close code blocks
* Update docs/source/en/model_doc/patchtst.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update tests/models/patchtst/test_modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/configuration_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/models/patchtst/modeling_patchtst.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* fix tests
* add add_start_docstrings
* move examples to the forward's docstrings
* update prepare_batch
* update test
* fix test_prediction_head
* fix generation test
* use seed to create generator
* add output_hidden_states and config.num_patches
* add loc and scale args in PatchTSTForPredictionOutput
* edit outputs if if not return_dict
* use self.share_embedding to check instead checking type.
* remove seed
* make style
* seed is an optional int
* fix test
* generator device
* Fix assertTrue test
* swap order of items in outputs when return_dict=False.
* add mask_type and random_mask_ratio to unittest
* Update modeling_patchtst.py
* add add_start_docstrings for regression model
* make style
* update model path
* Edit the ValueError comment in forecast_masking
* update examples
* make style
* fix commented code
* update examples: remove config from from_pretrained call
* Edit example outputs
* Set default target_values to None
* remove config setting in regression example
* Update configuration_patchtst.py
* Update configuration_patchtst.py
* remove config from examples
* change default d_model and ffn_dim
* norm_eps default
* set has_attentions to Trye and define self.seq_length = self.num_patche
* update docstring
* change variable mask_input to do_mask_input
* fix blank space.
* change logger.debug to logger.warning.
* remove unused PATCHTST_INPUTS_DOCSTRING
* remove all_generative_model_classes
* set test_missing_keys=True
* remove undefined params in the docstring.
---------
Co-authored-by: nnguyen <nnguyen@us.ibm.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nam Nguyen <namctin@gmail.com>
Co-authored-by: Wesley Gifford <79663411+wgifford@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Add `model_docs`
* Add
* Update Model adoc
* Update docs/source/ja/model_doc/bark.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/ja/model_doc/beit.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/ja/model_doc/bit.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/ja/model_doc/blenderbot.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/ja/model_doc/blenderbot-small.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* update reiew-1
* Update toctree.yml
* translating docs and fixes of PR #27401
* Update docs/source/ja/model_doc/bert.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/ja/model_doc/bert-generation.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update the model docs
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* docs: replace torch.distributed.run by torchrun
`transformers` now officially support pytorch >= 1.10.
The entrypoint `torchrun`` is present from 1.10 onwards.
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
* Update src/transformers/trainer.py
with @ArthurZucker's suggestion
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
---------
Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Fix mistral generate for long prompt / response
* Add unit test
* fix linter
* fix linter
* fix test
* add assisted generation test for mistral and load the model in 4 bit + fa2
* Successfully resolved the ZeroDivisionError exception in the utils.notebook.y file.
* Now I update little code mentioned by Peter
* Using Black package to reformat my file
* Now I using ruff libary to reformated my file
Change "convert predictions to logits" to "convert logits to
predictions" to fix semantic error in the evaluation section. Logits
need to be converted to predictions to evaluate the accuracy, not the
other way round
* Fix typo in warning message
The path of `default_cache_path` is hf_cache_home/hub. There is no
directory named transformers under hf_cache_home
* Fix a typo in comment
* Update the version number
v4.22.0 is the earlist version that contains those changes in PR #18492
* added flash attention for opt
* added to list
* fix use cache (#3)
* style fix
* fix text
* test fix2
* reverted until 689f599
* torch fx tests are working now!
* small fix
* added TODO docstring
* changes
* comments and .md file modification
---------
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* initial commit
* Add inital testing files and modify __init__ files to add UnivNet imports.
* Fix some bugs
* Add checkpoint conversion script and add references to transformers pre-trained model.
* Add UnivNet entries for auto.
* Add initial docs for UnivNet.
* Handle input and output shapes in UnivNetGan.forward and add initial docstrings.
* Write tests and make them pass.
* Write docs.
* Add UnivNet doc to _toctree.yml and improve docs.
* fix typo
* make fixup
* make fix-copies
* Add upsample_rates parameter to config and improve config documentation.
* make fixup
* make fix-copies
* Remove unused upsample_rates config parameter.
* apply suggestions from review
* make style
* Verify and add reason for skipped tests inherited from ModelTesterMixin.
* Add initial UnivNetGan integration tests
* make style
* Remove noise_length input to UnivNetGan and improve integration tests.
* Fix bug and make style
* Make UnivNet integration tests pass
* Add initial code for UnivNetFeatureExtractor.
* make style
* Add initial tests for UnivNetFeatureExtractor.
* make style
* Properly initialize weights for UnivNetGan
* Get feature extractor fast tests passing
* make style
* Get feature extractor integration tests passing
* Get UnivNet integration tests passing
* make style
* Add UnivNetGan usage example
* make style and use feature extractor from hub in integration tests
* Update tips in docs
* apply suggestions from review
* make style
* Calculate padding directly instead of using get_padding methods.
* Update UnivNetFeatureExtractor.to_dict to be UnivNet-specific.
* Update feature extractor to support using model(**inputs) and add the ability to generate noise and pad the end of the spectrogram in __call__.
* Perform padding before generating noise to ensure the shapes are correct.
* Rename UnivNetGan.forward's noise_waveform argument to noise_sequence.
* make style
* Add tests to test generating noise and padding the end for UnivNetFeatureExtractor.__call__.
* Add tests for checking batched vs unbatched inputs for UnivNet feature extractor and model.
* Add expected mean and stddev checks to the integration tests and make them pass.
* make style
* Make it possible to use model(**inputs), where inputs is the output of the feature extractor.
* fix typo in UnivNetGanConfig example
* Calculate spectrogram_zero from other config values.
* apply suggestions from review
* make style
* Refactor UnivNet conversion script to use load_state_dict (following persimmon).
* Rename UnivNetFeatureExtractor to UnivNetGanFeatureExtractor.
* make style
* Switch to using torch.tensor and torch.testing.assert_close for testing expected values/slices.
* make style
* Use config in UnivNetGan modeling blocks.
* make style
* Rename the spectrogram argument of UnivNetGan.forward to input_features, following Whisper.
* make style
* Improving padding documentation.
* Add UnivNet usage example to the docs.
* apply suggestions from review
* Move dynamic_range_compression computation into the mel_spectrogram method of the feature extractor.
* Improve UnivNetGan.forward return docstring.
* Update table in docs/source/en/index.md.
* make fix-copies
* Rename UnivNet components to have pattern UnivNet*.
* make style
* make fix-copies
* Update docs
* make style
* Increase tolerance on flaky unbatched integration test.
* Remove torch.no_grad decorators from UnivNet integration tests to try to avoid flax/Tensorflow test errors.
* Add padding_mask argument to UnivNetModel.forward and add batch_decode feature extractor method to remove padding.
* Update documentation and clean up padding code.
* make style
* make style
* Remove torch dependency from UnivNetFeatureExtractor.
* make style
* Fix UnivNetModel usage example
* Clean up feature extractor code/docstrings.
* apply suggestions from review
* make style
* Add comments for tests skipped via ModelTesterMixin flags.
* Add comment for model parallel tests skipped via the test_model_parallel ModelTesterMixin flag.
* Add # Copied from statements to copied UnivNetFeatureExtractionTest tests.
* Simplify UnivNetFeatureExtractorTest.test_batch_decode.
* Add support for unbatched padding_masks in UnivNetModel.forward.
* Refactor unbatched padding_mask support.
* make style
* [Whisper] Add seq gen
* [Whisper] Add seq gen
* more debug
* Fix whisper logit processor
* Improve whisper code further
* Fix more
* more debug
* more debug
* Improve further
* Add tests
* Prep for batch size > 1
* Get batch_size>1 working
* Correct more
* Add extensive tests
* more debug
* more debug
* more debug
* add more tests
* more debug
* Apply suggestions from code review
* more debug
* add comments to explain the code better
* add comments to explain the code better
* add comments to explain the code better
* Add more examples
* add comments to explain the code better
* fix more
* add comments to explain the code better
* add comments to explain the code better
* correct
* correct
* finalize
* Apply suggestions from code review
* Apply suggestions from code review
* Fix `resize_token_embeddings` about `requires_grad`
The method `resize_token_embeddings` should keep `requires_grad`
unchanged for all parameters in embeddings.
Previously, `resize_token_embeddings` always set `requires_grad`
to `True`. After fixed, `resize_token_embeddings` copy the
`requires_grad` attribute in the old embeddings.
