Minho Ryang
aaccf1844e
[docstring] Fix docstring for LlamaTokenizer and LlamaTokenizerFast ( #26669 )
...
* [docstring] Fix docstring for `LlamaTokenizer` and `LlamaTokenizerFast`
* [docstring] Fix docstring typo at `LlamaTokenizer` and `LlamaTokenizerFast`
2023-10-11 17:03:31 +02:00
Srijan Sahay Srivastava
245da7ed38
[Doctest] Add configuration_encoder_decoder.py ( #26519 )
...
* [Doctest] Add configuration_encoder_decoder.py
Added configuration_encoder_decoder.py to utils/documentation_tests.txt for doctest
* Revert "[Doctest] Add configuration_encoder_decoder.py"
This reverts commit bd653535a4
2023-10-03 11:21:24 +02:00
Adithya Hegde Kota
4b4c6aabfb
[Doctest] Add configuration_roformer.py ( #26530 )
...
* [Doctest] Add configuration_roformer.py
* [Doctest] Add configuration_roformer.py
* [Doctest] Add configuration_roformer.py
* [Doctest] Add configuration_roformer.py
* Removed documentation_test.txt
* Removed configuration_roformer.py
* Update not_doctested.txt
2023-10-02 17:19:13 +02:00
Lysandre Debut
78dd120282
Fix failing doctest ( #26450 )
...
* Fix doctest
* Adding modeling also for now
2023-09-27 18:47:26 +02:00
Chris Bamford
72958fcd3c
[Mistral] Mistral-7B-v0.1 support ( #26447 )
...
* [Mistral] Mistral-7B-v0.1 support
* fixing names
* slightly longer test
* fixups
* not_doctested
* wrongly formatted references
* make fixuped
---------
Co-authored-by: Timothee Lacroix <t@eugen.ai>
Co-authored-by: timlacroix <t@mistral.ai>
2023-09-27 18:30:46 +02:00
NielsRogge
ace74d16bd
Add Nougat ( #25942 )
...
* Add conversion script
* Add NougatImageProcessor
* Add crop margin
* More improvements
* Add docs, READMEs
* Remove print statements
* Include model_max_length
* Add NougatTokenizerFast
* Fix imports
* Improve postprocessing
* Improve image processor
* Fix image processor
* Improve normalize method
* More improvements
* More improvements
* Add processor, improve docs
* Simplify fast tokenizer
* Remove test file
* Fix docstrings
* Use NougatProcessor in conversion script
* Add is_levensthein_available
* Add tokenizer tests
* More improvements
* Use numpy instead of opencv
* Add is_cv2_available
* Fix cv2_available
* Add is_nltk_available
* Add image processor tests, improve crop_margin
* Add integration tests
* Improve integration test
* Use do_rescale instead of hacks, thanks Amy
* Remove random_padding
* Address comments
* Address more comments
* Add import
* Address more comments
* Address more comments
* Address comment
* Address comment
* Set max_model_input_sizes
* Add tests
* Add requires_backends
* Add Nougat to exotic tests
* Use to_pil_image
* Address comment regarding nltk
* Add NLTK
* Improve variable names, integration test
* Add test
* refactor, document, and test regexes
* remove named capture groups, add comments
* format
* add non-markdown fixed tokenization
* format
* correct flakyness of args parse
* add regex comments
* test functionalities for crop_image, align long axis and expected output
* add regex tests
* remove cv2 dependency
* test crop_margin equality between cv2 and python
* refactor table regexes to markdown
add newline
* change print to log, improve doc
* fix high count tables correction
* address PR comments: naming, linting, asserts
* Address comments
* Add copied from
* Update conversion script
* Update conversion script to convert both small and base versions
* Add inference example
* Add more info
* Fix style
* Add require annotators to test
* Define all keyword arguments explicitly
* Move cv2 annotator
* Add tokenizer init method
* Transfer checkpoints
* Add reference to Donut
* Address comments
* Skip test
* Remove cv2 method
* Add copied from statements
* Use cached_property
* Fix docstring
* Add file to not doctested
---------
Co-authored-by: Pablo Montalvo <pablo.montalvo.leroux@gmail.com>
2023-09-26 07:06:04 +02:00
Yih-Dar
c3ecf2d95d
Fix doctest CI ( #26324 )
...
fix doc CI
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-22 08:58:30 +02:00
Yih-Dar
f02b915ba2
Remove utils/documentation_tests.txt ( #26213 )
...
* update
* update
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-18 13:33:01 +02:00
Yih-Dar
d020a2b81b
No doctest for convert_bros_to_pytorch.py ( #26212 )
...
fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-09-18 13:31:59 +02:00
Arthur
9cccb3a838
[Persimmon] Add support for persimmon ( #26042 )
...
* intiial commit
* updates
* nits
* update conversion script
* update conversion script
* use path to load
* add tips etc
* some modeling logic
* modeling update
* more nits
* nits
* normal layer norm
* update config and doc
* nits
* update doc remove unused
* update
* fix inits and stuff
* fixup
* revert wrong changes
* updates
* more nits
* add default config values to the configuration file
* fixup happy
* update
* 2 tests left
* update readmes
* more nits
* slow test and more documentation
* update readme
* fix licences
* styling
* use fast if possible when saving tokenizer
* remove todo
* remove tokenization tests
* small last nits
* Apply suggestions from code review
Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
* nits to skip the timout doctest
* fix integration test
* fix test
* update eos token
* update to allow fast tokenization
* styling
* fix codeLlama as well for the update post processor
* Apply suggestions from code review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* add more copied from statements
* update
* doc passes doctest
* remove `# final layer norm?`
* change docstring prompot
* update
* Update README.md
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* don't doctest the conversion script as it requires more packages
* don't init a model in the config
* oups
* fix doctest
---------
Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2023-09-12 11:33:27 +02:00
Joao Gante
a35f889acc
Tests: detect lines removed from "utils/not_doctested.txt" and doctest ALL generation files ( #25763 )
2023-08-29 16:15:05 +01:00
Younes Belkada
4b79697865
🚨 🚨 🚨 [Refactor] Move third-party related utility files into integrations/ folder 🚨 🚨 🚨 ( #25599 )
...
* move deepspeed to `lib_integrations.deepspeed`
* more refactor
* oops
* fix slow tests
* Fix docs
* fix docs
* addess feedback
* address feedback
* final modifs for PEFT
* fixup
* ok now
* trigger CI
* trigger CI again
* Update docs/source/en/main_classes/deepspeed.md
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* import from `integrations`
* address feedback
* revert removal of `deepspeed` module
* revert removal of `deepspeed` module
* fix conflicts
* ooops
* oops
* add deprecation warning
* place it on the top
* put `FutureWarning`
* fix conflicts with not_doctested.txt
* add back `bitsandbytes` module with a depr warning
* fix
* fix
* fixup
* oops
* fix doctests
---------
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2023-08-25 17:13:34 +02:00
Joao Gante
494e96d8d6
Generate: logits processors are doctested and fix broken doctests ( #25692 )
...
* shorter example
* add logits processors to doctests
* remove file from conflict?
* tmp commit
* Fix broken tests; Shorter sampling tests
* Apply suggestions from code review
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-08-25 12:42:06 +01:00
Yih-Dar
b413e0610b
Remove utils/documentation_tests.txt ( #25680 )
...
* fix
* fix
* fix
* fix
* fix
* fix
* Apply suggestions from code review
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-08-23 11:14:45 +02:00
Yih-Dar
3d1edb6c5d
fix wrong path in some doc ( #25658 )
...
* update
* check
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-23 08:34:30 +02:00
Yih-Dar
2df24228d6
Skip doctest for some recent files ( #25631 )
...
update
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-21 15:20:44 +02:00
Yih-Dar
f09db47a71
Run doctest for new files ( #25588 )
...
fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-21 11:08:38 +02:00
