Arthur
f9cc333805
[ PreTrainedTokenizerFast] Keep properties from fast tokenizer ( #25053 )
...
* draft solution
* use `setdefault`
* nits
* add tests and fix truncation issue
* fix test
* test passes locally
* quality
* updates
* update tsets
2023-07-25 18:45:01 +02:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting ( #21480 )
...
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
2023-02-06 18:10:56 -05:00
SaulLu
ae7bae8fe7
fix train_new_from_iterator in the case of byte-level tokenizers ( #17549 )
2022-06-08 15:30:41 +02:00
Lysandre Debut
29c10a41d0
[Test refactor 1/5] Per-folder tests reorganization ( #15725 )
...
* Per-folder tests reorganization
Co-authored-by: sgugger <sylvain.gugger@gmail.com>
Co-authored-by: Stas Bekman <stas@stason.org>
2022-02-23 15:46:28 -05:00
