5 Commits

Author	SHA1	Message	Date
Ita Zaporozhets	3d7c2f9dea	#32184 save total_vocab_size (#32240) ... * save total_vocab_size = vocab_size + user added tokens to speed up operation * updating length when added_tokens_decoder is set * add test len(tokenizer)	2024-08-05 09:22:48 +02:00
Arthur	673440d073	update ruff version (#30932) ... * update ruff version * fix research projects * Empty * Fix errors --------- Co-authored-by: Lysandre <lysandre@huggingface.co>	2024-05-22 06:40:15 +02:00
Lysandre Debut	f497f564bb	Update all references to canonical models (#29001) ... * Script & Manual edition * Update	2024-02-16 08:16:58 +01:00
Sylvain Gugger	6f79d26442	Update quality tooling for formatting (#21480) ... * Result of black 23.1 * Update target to Python 3.7 * Switch flake8 to ruff * Configure isort * Configure isort * Apply isort with line limit * Put the right black version * adapt black in check copies * Fix copies	2023-02-06 18:10:56 -05:00
Lysandre Debut	29c10a41d0	[Test refactor 1/5] Per-folder tests reorganization (#15725) ... * Per-folder tests reorganization Co-authored-by: sgugger <sylvain.gugger@gmail.com> Co-authored-by: Stas Bekman <stas@stason.org>	2022-02-23 15:46:28 -05:00