Arthur
00b5887b94
🚨 🚨 🚨 [NLLB Tokenizer] Fix the prefix tokens 🚨 🚨 🚨 ( #22313 )
...
* fix the prefix tokens
* update fast and test values
* add legacy behaviour
Co-authored-by: sgugger <sylvain.gugger@gmail.com>
* update disclaimer, linkissue PR and behaviral changes
* Apply suggestions from code review
Co-authored-by: Lysandre Debut <hi@lysand.re>
* styling
* make a quote
* quote this time
---------
Co-authored-by: sgugger <sylvain.gugger@gmail.com>
Co-authored-by: Lysandre Debut <hi@lysand.re>
2023-04-04 14:53:06 +02:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting ( #21480 )
...
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
2023-02-06 18:10:56 -05:00
Sylvain Gugger
986526a0e4
Replace as_target context managers by direct calls ( #18325 )
...
* Preliminary work on tokenizers
* Quality + fix tests
* Treat processors
* Fix pad
* Remove all uses of in tests, docs and examples
* Replace all as_target_tokenizer
* Fix tests
* Fix quality
* Update examples/flax/image-captioning/run_image_captioning_flax.py
Co-authored-by: amyeroberts <amy@huggingface.co>
* Style
Co-authored-by: amyeroberts <amy@huggingface.co>
2022-07-29 08:09:09 -04:00
Lysandre Debut
c1c79b0655
NLLB tokenizer ( #18126 )
...
* NLLB tokenizer
* Apply suggestions from code review - Thanks Stefan!
Co-authored-by: Stefan Schweter <stefan@schweter.it>
* Final touches
* Style :)
* Update docs/source/en/model_doc/nllb.mdx
Co-authored-by: Stefan Schweter <stefan@schweter.it>
* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* PR reviews
* Auto models
Co-authored-by: Stefan Schweter <stefan@schweter.it>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2022-07-18 08:12:34 -04:00
