Joao Gante
89136ff7f8
Generate: sequence bias can handle same terminations ( #24822 )
2023-07-20 12:23:17 +01:00
Joao Gante
5f0801d174
Generate: add SequenceBiasLogitsProcessor ( #24334 )
2023-06-21 11:14:41 +01:00
Elad Segal
eec46b4f75
Fix MinNewTokensLengthLogitsProcessor when used with a list of eos tokens ( #21959 )
...
* Fix MinNewTokensLengthLogitsProcessor when used with a list of eos tokens
* fix docs
* Empty commit
* formatting
2023-03-07 11:59:22 +00:00
Karim Foda
b9403e9516
Add hallucination filter ( #18675 )
...
* Add hallucination penalty
* Make quality changes
* Inverse penalty
* Fix imports & quality
* Fix name spelling issue
* set encoder_repetition_penalty and fix quality
* Fix failing test
* Add to config_common_kwargs
* Fix modelling_rag error
* Update src/transformers/generation_logits_process.py
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
* Remove breakpoint
* Make style fixes
* Update encoder_repetition_penalty default value
* Merge latest main changes
* Make fixup changes
* Add EncoderRepetitionPenaltyLogitsProcessor to generation/__init__.py
* Fix repo-inconsistency
* Remove venv
* Remove tensorflow-macos & add tests
* Add documentation
* Fix quality issues
* move encoder_repetition_penalty to config
* Update src/transformers/configuration_utils.py
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
* Update src/transformers/generation/configuration_utils.py
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
* Remove encoder_repetition_penalty from tests
* Fix type error
* Fix format error
Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
2023-01-19 11:20:25 -05:00
Sherman Siu
865da84abb
Add Epsilon- and Eta-Sampling ( #21121 )
...
* Add epsilon- and eta-sampling.
Add epsilon- and eta-sampling, following the official code from https://github.com/john-hewitt/truncation-sampling and adapting to be more configurable, as required by Huggingface transformers.
* Add unit tests for epsilon- and eta-sampling.
* Black: fix code formatting.
* Fix docstring spacing.
* Clean up newlines.
* Fix implementation bugs and their associated tests.
* Remove epsilon- and eta-sampling parameters from PretrainedConfig.
* Clarify and clean up the documentation.
* Remove parameters for PretrainedConfig test.
2023-01-17 13:04:32 -05:00
Konstantin Kotik
367fdf3330
MinNewTokensLengthLogitsProcessor for .generate method #20814 ( #20892 )
...
* feat: add min new length logit processor
* test: add min new length logit processor
* docs: add MinNewTokensLengthLogitsProcessor
* feat: import MinNewTokensLengthLogitsProcessor
* fix: update pytorch dummy objects
* refactor & fix: rename attributes and var and get rid of dynamic attribute
* tests: align test with new interface
* docs: fix typo
* docs: minor clarification
* Empty-Commit
* empty commit
* run automated quality edits
Co-authored-by: Joao Gante <joao@huggingface.co>
2023-01-03 06:29:02 -05:00
Joao Gante
f270b960d6
Generate: move generation_*.py src files into generation/*.py ( #20096 )
...
* move generation_*.py src files into generation/*.py
* populate generation.__init__ with lazy loading
* move imports and references from generation.xxx.object to generation.object
2022-11-09 15:34:08 +00:00
