Motoki Wu 45da7cec5a Add custom stop token ids for generation (#20727) ... * Add StopIdStoppingCriteria * add a working test for stop id criteria * add to global scope * add stop_ids to generate * add pipeline test * use tokenizer encode in test * add test to generation utils * reformat * fixup * make-fix-copies * rename to stop_token_id * use stop_tokens instead * add to text to text generation * make fixup * make repo-consistency * Add support for list of ints for eos_token_id inside generation/utils.py * Instead of having if elses, cast the eos_token_id into a List[int] * Add List[int] support for logits_process.py * add List[int] for beam_search.py * add List[int] for forced_eos_token_id * revert stop token id stopping criteria changes * make fixup * fix tests * add eos_token_id to generation/utils.py and added tests test_utils.py * add eos_token_id type hints and fix for pad tokens * add comments * remove some prints and remove forced false test * fix * put back test_stop_sequence_stopping_criteria * remove unused import and make fixup * add a none check * update docstring * add more docstring for list ints * make fixup		2023-01-03 15:18:24 -05:00
..
__init__.py	[Test refactor 1/5] Per-folder tests reorganization (#15725)	2022-02-23 15:46:28 -05:00
test_beam_constraints.py	Generate: move generation_*.py src files into generation/*.py (#20096)	2022-11-09 15:34:08 +00:00
test_beam_search.py	Fix tiny typo (#20841)	2022-12-20 03:17:59 -05:00
test_configuration_utils.py	Generate: use GenerationConfig as the basis for .generate() parametrization (#20388)	2022-12-15 18:27:20 +00:00
test_flax_logits_process.py	Generate: move generation_*.py src files into generation/*.py (#20096)	2022-11-09 15:34:08 +00:00
test_flax_utils.py	Generate: move generation_*.py src files into generation/*.py (#20096)	2022-11-09 15:34:08 +00:00
test_logits_process.py	MinNewTokensLengthLogitsProcessor for .generate method #20814 (#20892)	2023-01-03 06:29:02 -05:00
test_stopping_criteria.py	Generate: move generation_*.py src files into generation/*.py (#20096)	2022-11-09 15:34:08 +00:00
test_tf_logits_process.py	Generate: move generation_*.py src files into generation/*.py (#20096)	2022-11-09 15:34:08 +00:00
test_tf_utils.py	Generate: move generation_*.py src files into generation/*.py (#20096)	2022-11-09 15:34:08 +00:00
test_utils.py	Add custom stop token ids for generation (#20727)	2023-01-03 15:18:24 -05:00