3 Commits

Author	SHA1	Message	Date
cyyever	1e6b546ea6	Use Python 3.9 syntax in tests (#37343) ... Signed-off-by: cyy <cyyever@outlook.com>	2025-04-08 14:12:08 +02:00
Yih-Dar	1fcaad6df9	Use lru_cache for tokenization tests (#36818) ... * fix * fix * fix * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2025-03-28 15:09:35 +01:00
Arie Pratama Sutiono	f2bf4fcf3d	Add SplinterTokenizer unit test (#32652) ... * add unit tests for splinter_tokenizer * add unit test for splinter tokenizer, pass in the question_token to be saved on save_pretrained called * remove unused import * remove vocab_splinter.txt, add Copied from, use fmt:on and fmt:off to prevent autoformatting on long lines * remove all the spaces Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>	2024-10-03 14:49:56 +02:00