transformers

mirror of https://github.com/huggingface/transformers.git synced 2025-07-17 19:48:23 +06:00

Author	SHA1	Message	Date
Phuc Van Phan	656e17f6f7	correct resume training steps number in progress bar (#25691 ) feat: correct update resume update with steps	2023-08-23 20:09:14 +02:00
Matt	62396cff46	TF 2.14 compatibility (#25630 ) * Update the TF pin and see if anything breaks * make fixup * make fixup * make fixup	2023-08-22 13:13:38 +01:00
Sylvain Gugger	5c67682b16	v4.33.0.dev0	2023-08-21 07:07:04 -04:00
Zach Mueller	ca51499248	Make training args fully immutable (#25435 ) * Make training args fully immutable * Working tests, PyTorch * In test_trainer * during testing * Use proper dataclass way * Fix test * Another one * Fix tf * Lingering slow * Exception * Clean	2023-08-15 11:47:47 -04:00
dependabot[bot]	7a94ea4c64	Bump tornado from 6.3.2 to 6.3.3 in /examples/research_projects/lxmert (#25511 ) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3) --- updated-dependencies: - dependency-name: tornado dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-08-15 08:52:30 +02:00
dependabot[bot]	2552b8c5bd	Bump tornado from 6.3.2 to 6.3.3 in /examples/research_projects/visual_bert (#25512 ) Bump tornado in /examples/research_projects/visual_bert Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3) --- updated-dependencies: - dependency-name: tornado dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-08-15 08:52:20 +02:00
dependabot[bot]	0ebe7ae160	Bump gitpython from 3.1.30 to 3.1.32 in /examples/research_projects/decision_transformer (#25467 ) Bump gitpython in /examples/research_projects/decision_transformer Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.32. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.32) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-08-13 19:47:16 +02:00
dependabot[bot]	2b22cde71e	Bump gitpython from 3.1.30 to 3.1.32 in /examples/research_projects/distillation (#25468 ) Bump gitpython in /examples/research_projects/distillation Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.32. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.32) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-08-13 19:47:04 +02:00
Gema Parreño	0acf56224b	Update run_translation.py broken link example Pytoch (#25461 ) * Update run_translation.py Fixed link * Update run_translation.py	2023-08-11 15:41:24 +02:00
Yih-Dar	9c7b744795	Fix missing usage of `token` (#25382 ) * add missing tokens * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-08-08 16:27:24 +02:00
Zach Mueller	01ab39b65f	Load state in else (#25318 ) * Load else * New approach * Propagate	2023-08-08 05:41:00 -04:00
Phuc Van Phan	5fe36970e5	Adding more information in help parser on train_file and validation_file (#25324 ) chorse: adding new doc on train and val	2023-08-07 17:56:13 +02:00
Jackmin801	145109382a	Allow `trust_remote_code` in example scripts (#25248 ) * pytorch examples * pytorch mim no trainer * cookiecutter * flax examples * missed line in pytorch run_glue * tensorflow examples * tensorflow run_clip * tensorflow run_mlm * tensorflow run_ner * tensorflow run_clm * pytorch example from_configs * pytorch no trainer examples * Revert "tensorflow run_clip" This reverts commit `261f86ac1f`. * fix: duplicated argument	2023-08-07 16:32:25 +02:00
Yih-Dar	149cb0cce2	Add `token` arugment in example scripts (#25172 ) * fix * fix * fix * fix * fix * fix * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-08-02 11:17:31 +02:00
Yih-Dar	d53b8ad780	Update `use_auth_token` -> `token` in example scripts (#25167 ) * pytorch examples * tensorflow examples * flax examples --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-07-28 15:33:45 +02:00
Alan Ji	afa96fffdf	make run_generation more generic for other devices (#25133 ) * make run_generation more generic for other devices * use Accelerate to support any device type it supports. * make style * fix error usage of accelerator.prepare_model * use `PartialState` to make sure everything is running on the right device --------- Co-authored-by: statelesshz <jihuazhong1@huawei.com>	2023-07-28 08:20:10 -04:00
Lucain	6232c380f2	Fix `.push_to_hub` and cleanup `get_full_repo_name` usage (#25120 ) * Fix .push_to_hub and cleanup get_full_repo_name usage * Do not rely on Python bool conversion magic * request changes	2023-07-28 11:40:08 +02:00
dependabot[bot]	f1deb21fce	Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/visual_bert (#25097 ) Bump certifi in /examples/research_projects/visual_bert Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-25 17:25:14 -04:00
dependabot[bot]	45bde362d2	Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/decision_transformer (#25098 ) Bump certifi in /examples/research_projects/decision_transformer Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-25 17:25:05 -04:00
dependabot[bot]	6b8dbc283c	Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/lxmert (#25096 ) Bump certifi in /examples/research_projects/lxmert Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-25 17:24:50 -04:00
Alan Ji	c879318cc5	replace `per_gpu_eval_batch_size` with `per_device_eval_batch_size` in readme of multiple-choice task (#25078 ) replace `per_gpu_eval_batch_size` with `per_device_eval_batch_size` in readme of multiple-choice	2023-07-25 08:11:56 -04:00
Zach Mueller	aa1b09c5d1	Change logic for logging in the examples (#24956 ) Change logic	2023-07-20 12:30:10 -04:00
dependabot[bot]	9f912ef62a	Bump aiohttp from 3.8.1 to 3.8.5 in /examples/research_projects/decision_transformer (#24954 ) Bump aiohttp in /examples/research_projects/decision_transformer Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.8.1 to 3.8.5. - [Release notes](https://github.com/aio-libs/aiohttp/releases) - [Changelog](https://github.com/aio-libs/aiohttp/blob/v3.8.5/CHANGES.rst) - [Commits](https://github.com/aio-libs/aiohttp/compare/v3.8.1...v3.8.5) --- updated-dependencies: - dependency-name: aiohttp dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-20 12:17:38 -04:00
dependabot[bot]	9859806608	Bump pygments from 2.11.2 to 2.15.0 in /examples/research_projects/decision_transformer (#24949 ) Bump pygments in /examples/research_projects/decision_transformer Bumps [pygments](https://github.com/pygments/pygments) from 2.11.2 to 2.15.0. - [Release notes](https://github.com/pygments/pygments/releases) - [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES) - [Commits](https://github.com/pygments/pygments/compare/2.11.2...2.15.0) --- updated-dependencies: - dependency-name: pygments dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-20 07:43:48 -04:00
statelesshz	37d8611ac9	replace no_cuda with use_cpu in test_pytorch_examples (#24944 ) * replace no_cuda with use_cpu in test_pytorch_examples * remove codes that never be used * fix style	2023-07-20 07:09:04 -04:00
ranchlai	8fd8c8e49e	Add multi-label text classification support to pytorch example (#24770 ) * Add text classification example * set the problem type and finetuning task * ruff reformated * fix bug for unseting label_to_id for regression * update README.md * fixed finetuning task * update comment * check if label exists in feature before removing * add useful logging	2023-07-20 07:02:44 -04:00
Hwijeen Ahn	dd49404a89	check if eval dataset is dict (#24877 ) * check if eval dataset is dict * formatting	2023-07-18 13:33:41 -04:00
Sylvain Gugger	e9ad51306f	4.32.0.dev0	2023-07-17 13:30:44 -04:00
dependabot[bot]	38dfb86958	Bump cryptography from 41.0.0 to 41.0.2 in /examples/research_projects/decision_transformer (#24833 ) Bump cryptography in /examples/research_projects/decision_transformer Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.0 to 41.0.2. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/41.0.0...41.0.2) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-17 07:17:17 -04:00
Bauke Brenninkmeijer	fc9e387dc0	Replacement of 20 asserts with exceptions (#24757 ) * initial replacements of asserts with errors/exceptions * replace assert with exception in generation, align and bart * reset formatting change * reset another formatting issue * Apply suggestion Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * don't touch this file * change to 'is not False' * fix type --------- Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2023-07-12 07:45:09 -04:00
Gema Parreño	4b26a61631	Fix loading dataset docs link in run_translation.py example (#24594 ) * fix loading dataset link * Update examples/tensorflow/translation/run_translation.py Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> * Update examples/tensorflow/translation/run_translation.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> --------- Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>	2023-07-03 15:21:21 +01:00
Jeroen Van Goey	fd8dcd0953	Udate link to RunHouse hardware setup documentation. (#24590 ) * Udate link to RunHouse hardware setup documentation. * Fix link to hardware setup in other location as well	2023-06-30 12:11:58 +01:00
Xiaoli Wang	239ace152b	Fix TypeError: Object of type int64 is not JSON serializable (#24340 ) * Fix TypeError: Object of type int64 is not JSON serializable * Convert numpy.float64 and numpy.int64 to float and int for json serialization * Black reformatted examples/pytorch/token-classification/run_ner_no_trainer.py * * make style	2023-06-27 12:15:49 +01:00
Matt	8e164c5400	Improved keras imports (#24448 ) * An end to accursed version-specific imports * No more K.is_keras_tensor() either * Update dependency tables * Use a cleaner call context function getter * Add a cap to <2.14 * Add cap to examples requirements too	2023-06-23 19:09:34 +01:00
Patrick von Platen	1609a436ec	Add MMS CTC Fine-Tuning (#24281 ) * Add mms ctc fine tuning * make style * More fixes that are needed * make fix-copies * make draft for README * add new file * move to new file * make style * make style * add quick test * make style * make style	2023-06-15 01:10:27 +02:00
Ethan	f7d80cb3d2	Fix steps bugs in no trainer examples (#24197 ) Fix step bugs in no trainer + load checkpoint + grad acc	2023-06-12 11:49:55 -04:00
Sylvain Gugger	ba695c1efd	v4.31.0.dev0	2023-06-07 16:49:00 -04:00
Zachary Mueller	cbf6bc2350	Oops, missed one (#24054 ) Oops	2023-06-06 13:30:19 -04:00
Zachary Mueller	072188d638	Act on deprecations in Accelerate no_trainer examples (#24053 ) Act on deprecation	2023-06-06 13:04:38 -04:00
dependabot[bot]	539e2281cd	Bump cryptography from 39.0.1 to 41.0.0 in /examples/research_projects/decision_transformer (#23964 ) Bump cryptography in /examples/research_projects/decision_transformer Bumps [cryptography](https://github.com/pyca/cryptography) from 39.0.1 to 41.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/39.0.1...41.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-06-02 16:23:44 -04:00
Matt	167a0d8f87	Add an option to reduce compile() console spam (#23938 ) * Add an option to reduce compile() console spam * Add annotations to the example scripts * Add notes to the quicktour docs as well * minor fix	2023-06-02 15:28:52 +01:00
Sylvain Gugger	3ff443a6d9	Re-enable squad test (#23912 ) * Re-enable squad test * [all-test] * [all-test] Fix all test command * Fix the all-test	2023-05-31 13:44:26 -04:00
Sylvain Gugger	00f6ba0e7e	Skip failing test for now	2023-05-31 06:31:33 -04:00
Ran Ran	e724246935	Fix no such file or directory error (#23783 ) * Fix no such file or directory error * Address comment * Fix formatting issue	2023-05-26 14:24:57 -04:00
dependabot[bot]	d685e330b5	Bump tornado from 6.0.4 to 6.3.2 in /examples/research_projects/visual_bert (#23767 ) Bump tornado in /examples/research_projects/visual_bert Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.0.4 to 6.3.2. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.0.4...v6.3.2) --- updated-dependencies: - dependency-name: tornado dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-25 16:16:12 -04:00
dependabot[bot]	4b0e7ded1c	Bump tornado from 6.0.4 to 6.3.2 in /examples/research_projects/lxmert (#23766 ) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.0.4 to 6.3.2. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.0.4...v6.3.2) --- updated-dependencies: - dependency-name: tornado dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-25 16:16:01 -04:00
Sylvain Gugger	6e4bc67099	Revamp test selection for the example tests (#23737 ) * Revamp test selection for the example tests * Rename old XLA test and fake modif in run_glue * Fixes * Fake Trainer modif * Remove fake modifs	2023-05-25 09:38:21 -04:00
Wang, Yi	33687a3f61	add GPTJ/bloom/llama/opt into model list and enhance the jit support (#23291 ) Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>	2023-05-24 10:57:56 +01:00
zspo	003a0cf8cc	Fix some docs what layerdrop does (#23691 ) * Fix some docs what layerdrop does * Update src/transformers/models/data2vec/configuration_data2vec_audio.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Fix more docs --------- Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2023-05-23 14:50:40 -04:00
dependabot[bot]	aa30cd4f3f	Bump requests from 2.22.0 to 2.31.0 in /examples/research_projects/lxmert (#23668 ) Bump requests in /examples/research_projects/lxmert Bumps [requests](https://github.com/psf/requests) from 2.22.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](https://github.com/psf/requests/compare/v2.22.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-23 05:31:53 -04:00
dependabot[bot]	9bf72ae564	Bump requests from 2.22.0 to 2.31.0 in /examples/research_projects/visual_bert (#23670 ) Bump requests in /examples/research_projects/visual_bert Bumps [requests](https://github.com/psf/requests) from 2.22.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](https://github.com/psf/requests/compare/v2.22.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-23 05:31:30 -04:00
dependabot[bot]	ecc05f8c1e	Bump requests from 2.27.1 to 2.31.0 in /examples/research_projects/decision_transformer (#23673 ) Bump requests in /examples/research_projects/decision_transformer Bumps [requests](https://github.com/psf/requests) from 2.27.1 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](https://github.com/psf/requests/compare/v2.27.1...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-23 05:28:09 -04:00
Zachary Mueller	b191d7db44	Update all no_trainer with skip_first_batches (#23664 )	2023-05-22 14:49:31 -04:00
sshahrokhi	6f72e71f97	changing the requirements to a cpu torch version that works (#23483 )	2023-05-22 12:58:55 -04:00
Boda Sadallah	a7920065f2	fix bug in group_texts function, that was inserting short batches (#23429 ) * fix bug in group_texts function, that was inserting short batches * fully exclude short batches and return empty dict instead * fix style	2023-05-18 14:22:30 -04:00
Zachary Mueller	8a58809312	Fix translation no_trainer (#23407 ) * Fix translation	2023-05-16 13:10:42 -04:00
Yih-Dar	d51296d9c2	skip `test_run_squad_no_trainer` for now (#23302 ) skip Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-05-11 19:26:48 +02:00
Hari	5d02e6bd20	Convert numpy arrays to lists before saving the evaluation metrics as json (#23268 ) * convert numpy array to list before writing to json per_category_iou and per_category_accuracy are ndarray in the eval_metrics * code reformatted with make style	2023-05-11 08:54:23 -04:00
Maria Khalusova	91f4c84a19	CTC example: updated trainer parameters to save tokenizer (#23243 ) trainer parameters changed to save tokenizer in addition to feature_extractor	2023-05-10 07:45:10 -04:00
Sylvain Gugger	a0c0a78233	v4.30.0.dev0	2023-05-09 14:59:38 -04:00
Nicolas Patry	c34a525d2f	Proposed fix for TF example now running on safetensors. (#23208 ) * Proposed fix for TF example now running on safetensors. * Adding more warnings and returning keys. * Trigger CI * Trigger CI --------- Co-authored-by: Sylvain Gugger <Sylvain.gugger@gmail.com>	2023-05-09 13:04:27 -04:00
Sebastian	1a8f61110e	fix: Update run_qa.py to work with deepset/germanquad (#23225 ) Call str on id to make sure any ints are converted into the expected format for squad datasets	2023-05-09 09:20:10 -04:00
Sylvain Gugger	fd6970bc56	Skip failing test	2023-05-08 08:52:44 -04:00
Ashwin Mathur	fc6c8b0eaa	Add `no_trainer` scripts to pre-train Vision Transformers (#23156 ) * Add run_mim_no_trainer.py draft from #20412 Add parse_args method and copy over other dependencies Add Method call for sending telemetry Initialize Accelerator Make one log on every process Set seed and Handle repository creation Initialize dataset and Set validation split Create Config Adapt Config Update Config Create Feature Extractor Create model Set column names Create transforms Create mask generator Create method to preprocess images Shuffle datasets if needed and set transforms Create Dataloaders Add optimizer Add learning rate scheduler Prepare everything with our accelerator Tie weights for TPU training Recalculate training steps and training epochs Set accelerator checkpointing steps Initialize trackers and store configuration Set total batch size Fix typo: mlm -> mim Log info at the start of training Load in the weights and states from previous save update the progress_bar if load from checkpoint Define train loop Add evaluation loop to training Add to parse_args method Push repo to hub Save accelerator state End training and save model and feature extractor Remove unused imports Fix trailing whitespace * Update code based on comments, Rename feature_extractor to image_processor * Fix linting * Add argument for learning rate * Add argument for setting number of training epochs * Remove incorrect logger argument * Convert max_train_steps to int for tqdm --------- Co-authored-by: Saad Mahmud <shuvro.mahmud79@gmail.com>	2023-05-05 13:22:49 -04:00
Robert Stone	b6933d76d2	Tidy Pytorch GLUE benchmark example (#23134 ) Migration to Evaluate for metric is not quite complete	2023-05-03 15:50:41 -04:00
Alex Punnen	805db1fe13	num_noise_spans should be <= num_items #22246 (#22938 )	2023-05-02 13:07:30 -04:00
regisss	bcedd0a471	Save the tokenizer and image preprocessor after training a model with the contrastive image-text example (#23035 ) Save tokenizer and image preprocessor	2023-05-02 09:23:16 -04:00
dependabot[bot]	b8648290d2	Bump flask from 2.0.3 to 2.3.2 in /examples/research_projects/decision_transformer (#23094 ) Bump flask in /examples/research_projects/decision_transformer Bumps [flask](https://github.com/pallets/flask) from 2.0.3 to 2.3.2. - [Release notes](https://github.com/pallets/flask/releases) - [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/flask/compare/2.0.3...2.3.2) --- updated-dependencies: - dependency-name: flask dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-01 20:15:11 -04:00
Lingepumpe	5427250351	Avoid invalid escape sequences, use raw strings (#22936 ) * Avoid invalid escape sequences, use raw strings * Integrate PR feedback	2023-04-25 09:17:56 -04:00
Roy Hvaara	874c7caf19	Remove broken test_data symlink in legacy s2s examples (#22876 )	2023-04-21 15:35:42 +01:00
Sayak Paul	4116d1ec75	[Examples/TensorFlow] minor refactoring to allow compatible datasets to work (#22879 ) minor refactoring to allow compatible datasets to work.	2023-04-20 18:21:01 +05:30
Zachary Mueller	cd3e0211a6	Remove accelerate from tf test reqs (#22777 ) Remove accelerate from tf	2023-04-17 12:31:21 -04:00
Matt	2237127a6c	Fix sneaky torch dependency in TF example (#22804 )	2023-04-17 16:11:52 +01:00
Sayak Paul	390e121fb5	[Examples] TPU-based training of a language model using TensorFlow (#21657 ) * add: tokenizer training script for TF TPU LM training. * add: script for preparing the TFRecord shards. * add: sequence of execution to readme. * remove limit from the tfrecord shard name. * Add initial train_model.py * Add basic training arguments and model init * Get up to the point of writing the data collator * Pushing progress so far! * Complete first draft of model training code * feat: grouping of texts efficiently. Co-authored-by: Matt <rocketknight1@gmail.com> * Add proper masking collator and get training loop working * fix: things. * Read sample counts from filenames * Read sample counts from filenames * Draft README * Improve TPU warning * Use distribute instead of distribute.experimental * Apply suggestions from code review Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> * Modularize loading and add MLM probability as arg * minor refactoring to better use the cli args. * readme fillup. * include tpu and inference sections in the readme. * table of contents. * parallelize maps. * polish readme. * change script name to run_mlm.py * address PR feedback (round I). --------- Co-authored-by: Matt <rocketknight1@gmail.com> Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>	2023-04-14 10:41:01 +05:30
Sylvain Gugger	888c4a2ae0	v4.29.0.dev0	2023-04-12 20:04:29 -04:00
Sylvain Gugger	1b1867d86b	Replace -100s in predictions by the pad token (#22693 ) * Replace -100s in predictions by the pad token * Style * Try to catch them all	2023-04-11 09:32:20 -04:00
Mikel Penagarikano	d5239bab5b	Sync preprocesses before loading the processor at run_speech_recognition_ctc.py (#21926 ) * Update run_speech_recognition_ctc.py Make sure all processes wait until data is saved before loading the processor from the output_dit * Make sure all processes wait until data is saved before loading the processor from the output_dit * Update run_speech_recognition_ctc.py * Update run_speech_recognition_seq2seq.py	2023-04-05 09:36:04 -04:00
Maziyar Panahi	98268b2e76	Add id2label and label2id to model's config in run_xnil (#22558 ) Add id2label and label2id to config in run_xnil	2023-04-04 09:28:57 -04:00
dependabot[bot]	6fc44656b4	Bump redis from 4.5.3 to 4.5.4 in /examples/research_projects/decision_transformer (#22494 ) Bump redis in /examples/research_projects/decision_transformer Bumps [redis](https://github.com/redis/redis-py) from 4.5.3 to 4.5.4. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](https://github.com/redis/redis-py/compare/v4.5.3...v4.5.4) --- updated-dependencies: - dependency-name: redis dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-03-31 10:50:33 -04:00
Sabine	173193ccd0	Update Neptune docs (#22452 )	2023-03-29 13:15:38 -04:00
dependabot[bot]	32ff06403d	Bump redis from 4.1.4 to 4.5.3 in /examples/research_projects/decision_transformer (#22410 ) Bump redis in /examples/research_projects/decision_transformer Bumps [redis](https://github.com/redis/redis-py) from 4.1.4 to 4.5.3. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](https://github.com/redis/redis-py/compare/v4.1.4...v4.5.3) --- updated-dependencies: - dependency-name: redis dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-03-27 20:23:55 -04:00
Sylvain Gugger	057e1d7473	Fix quality	2023-03-27 13:17:14 -04:00
Donny Greenberg	f02e3a2b18	Hardware Auto-Setup for Examples (#22319 ) * Add initial remote hardware auto-setup docs * Fix a few typos and clarify some language * Add missing dependency * Update self-hosted launch script with Sylvain's comments. * Formatting. * Trigger CI * Style	2023-03-27 13:07:53 -04:00
Joao Gante	88dae78f4d	TensorFlow: pin maximum version to 2.12 (#22364 )	2023-03-24 18:45:03 +00:00
Sylvain Gugger	6587125c0a	Pin tensorflow-text to go with tensorflow (#22362 ) * Pin tensorflow-text to go with tensorflow * Make it more convenient to pin TensorFlow * setup don't like f-strings	2023-03-24 10:54:06 -04:00
Sylvain	ef28df0572	Fix quality due to ruff release	2023-03-22 20:45:08 -04:00
Connor Henderson	8e6c34b390	fix: Allow only test_file in pytorch and flax summarization (#22293 ) allow only test_file in pytorch and flax summarization	2023-03-22 10:46:56 +00:00
Wang, Yi	4ccaf268fb	add low_cpu_mem_usage option in run_clm.py example which will benefit… (#22288 ) * add low_cpu_mem_usage option in run_clm.py example which will benefit LLM loading Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * update all the example and README under language-modeling Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> --------- Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>	2023-03-22 10:42:39 +00:00
jiqing-feng	8472a224fb	Enable traced model for text-generation task (#22265 )	2023-03-22 10:19:26 +00:00
Sylvain Gugger	ebdb185bef	v4.28.0.dev0	2023-03-14 13:49:10 -04:00
bofeng huang	6192549c1f	[examples/speech-recognition] Add SpecAugment to run_speech_recognition_seq2seq.py (#21942 ) * Add specaugment to run_speech_recognition_seq2seq.py * Remove useless argument: text_column * Fix quality * Update return_attention_mask condition * Update specaugment arguments only for whisper models * Remove SpecAugment arguments from ModelArguments, only leave default values for simplicity * Apply suggestions from code review Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * Update apply_spec_augment only for whisper models * Apply suggestions from code review Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * Rename return_attention_mask to forward_attention_mask to avoid confusion with wav2vec2 models --------- Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>	2023-03-08 17:59:31 +01:00
Matt	d128f2ffab	Stop requiring Torch for our TF examples! (#21997 ) * Stop requiring Torch for our TF examples! * Slight tweak to logging in the example itself	2023-03-07 15:54:10 +00:00
Matt	5d8efc79db	Add TF contrastive image text finetuning example (#21939 ) * Initial commit * stash commit * Add model checkpointing and pushing * Fix model name inference * Update README * Update README * Remove a couple of Torch references * Update copyright date * make fixup * Update PushToHubCallback args! * Remove the torch summary * Add strategy.scope	2023-03-06 16:57:40 +00:00
Matt	1d3a1cc44b	Add check for different embedding types in examples (#21881 ) * Add check for different embedding types in examples * Correctly update summarization example	2023-03-01 16:57:06 +00:00
bofeng huang	3c0ce60855	[examples/summarization] deal with `max_length` and `num_beams` (#21740 ) * Override the decoding parameters of Seq2SeqTrainer * Fix quality * Fix max_length parameter * Fix quality * Remove redundant parameter max_length * Separate the preprocess of train and validation to use different max_target_length	2023-02-27 08:18:14 +01:00
Sanchit Gandhi	13489248fa	[Examples] Generalise run audio classification for log-mel models (#21756 ) * [Examples] Generalise run audio classification for log-mel models * batch feature extractor * make style	2023-02-24 09:19:07 +01:00
Sylvain Gugger	b19d64d852	Respect documentation on passive log level (#21700 ) * Respect documentation on passive log level * Fix test and set log level in examples * Add doc	2023-02-22 09:39:18 +01:00
Aaron Gokaslan	5e8c8eb5ba	Apply ruff flake8-comprehensions (#21694 )	2023-02-22 09:14:54 +01:00
Arthur	4194e5f42b	Fix-rag-finetune-project-requirement (#21697 ) pin pytorch lightning requirement	2023-02-20 17:23:39 +01:00
dependabot[bot]	fcfd4ec789	Bump werkzeug from 2.0.3 to 2.2.3 in /examples/research_projects/decision_transformer (#21658 ) Bump werkzeug in /examples/research_projects/decision_transformer Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.0.3 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/2.0.3...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-02-16 09:23:43 -05:00
regisss	751f17aa48	Fix typos in contrastive-image-text example README (#21665 )	2023-02-16 09:10:25 -05:00
Warren Green	fd5320bb57	Add missing arguemtn to run_clip.py (#21588 )	2023-02-13 10:27:23 -05:00
dependabot[bot]	92487f5d0b	Bump ipython from 8.1.1 to 8.10.0 in /examples/research_projects/decision_transformer (#21577 ) Bump ipython in /examples/research_projects/decision_transformer Bumps [ipython](https://github.com/ipython/ipython) from 8.1.1 to 8.10.0. - [Release notes](https://github.com/ipython/ipython/releases) - [Commits](https://github.com/ipython/ipython/compare/8.1.1...8.10.0) --- updated-dependencies: - dependency-name: ipython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-02-13 10:21:50 -05:00
steventk-g	c88b11c591	Add _mp_fn to run_mae.py for XLA testing (#21551 ) Update run_mae.py	2023-02-10 09:53:55 -05:00
lee1jun	b31cee6727	fix typo in run_speech_recognition_ctc.py (#21528 ) Update run_speech_recognition_ctc.py There should be `# limitations under the License` line at the end of the documentation section.	2023-02-09 09:46:40 -05:00
Stefan Schweter	d3046dad80	[Doc] Minor URL fixes in PyTorch Text Classification Readme (#21511 ) docs: fix some references in PyTorch text classification readme	2023-02-08 09:39:52 -05:00
dependabot[bot]	e024cd715e	Bump cryptography from 36.0.2 to 39.0.1 in /examples/research_projects/decision_transformer (#21507 ) Bump cryptography in /examples/research_projects/decision_transformer Bumps [cryptography](https://github.com/pyca/cryptography) from 36.0.2 to 39.0.1. - [Release notes](https://github.com/pyca/cryptography/releases) - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/36.0.2...39.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-02-08 09:25:06 -05:00
Sylvain Gugger	67d074874d	Cleanup quality (#21493 ) * Remove mentions of flake8/isort * Clean up inits * Deall with all other inits * Last special rule for dummy files	2023-02-07 12:27:31 -05:00
Jeroen Van Der Donckt	bbe98ea9c3	🖊️ fix typo in pytorch semantic segmentation readme (#21492 )	2023-02-07 09:39:24 -05:00
dependabot[bot]	35f93f299f	Bump oauthlib from 3.2.1 to 3.2.2 in /examples/research_projects/decision_transformer (#21481 ) Bump oauthlib in /examples/research_projects/decision_transformer Bumps [oauthlib](https://github.com/oauthlib/oauthlib) from 3.2.1 to 3.2.2. - [Release notes](https://github.com/oauthlib/oauthlib/releases) - [Changelog](https://github.com/oauthlib/oauthlib/blob/master/CHANGELOG.rst) - [Commits](https://github.com/oauthlib/oauthlib/compare/v3.2.1...v3.2.2) --- updated-dependencies: - dependency-name: oauthlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-02-06 18:27:14 -05:00
Sylvain Gugger	6f79d26442	Update quality tooling for formatting (#21480 ) * Result of black 23.1 * Update target to Python 3.7 * Switch flake8 to ruff * Configure isort * Configure isort * Apply isort with line limit * Put the right black version * adapt black in check copies * Fix copies	2023-02-06 18:10:56 -05:00
Stas Bekman	3b9a1dc132	[examples] improve block_size warning message (#21463 )	2023-02-06 08:36:12 -08:00
Kaustubh Dhole	182afb7dc6	Fixed RAG script which was failing on dummy example (#21416 ) * do not use prefix="val" for test The dummy example fails when test_epoch_end is called. The prefix="test" should be dynamic in the log metrics too. * Create test.source * Create test.target	2023-02-06 09:27:34 -05:00
Erwann Millon	ea55bd86b9	Add VQGAN-CLIP research project (#21329 ) * Add VQGAN-CLIP research project * fixed style issues * Update examples/research_projects/vqgan-clip/README.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/VQGAN_CLIP.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/requirements.txt Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/README.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/VQGAN_CLIP.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/VQGAN_CLIP.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/VQGAN_CLIP.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/loaders.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * replace CLIPProcessor with tokenizer, change asserts to exceptions * rm unused import * remove large files (jupyter notebook linked in readme, imgs migrated to hf dataset) * add tokenizers dependency * Remove comment Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * rm model checkpoints --------- Co-authored-by: Erwann Millon <erwann@Erwanns-MacBook-Air.local> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>	2023-02-02 14:45:35 -05:00
amyeroberts	e5db7051a8	Add TF image classification example script (#19956 ) * TF image classification script * Update requirements * Fix up * Add tests * Update test fetcher Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Fix directory path * Adding `zero-shot-object-detection` pipeline doctest. (#20274) * Adding `zero-shot-object-detection` pipeline doctest. * Remove nested_simplify. * Add generate kwargs to `AutomaticSpeechRecognitionPipeline` (#20952) * Add generate kwargs to AutomaticSpeechRecognitionPipeline * Add test for generation kwargs * Trigger CI * Data collator returns np * Update feature extractor -> image processor * Bug fixes - updates to reflect changes in API * Update flags to match PT & run faster * Update instructions - Maria's comment * Update examples/tensorflow/image-classification/README.md * Remove slow decorator --------- Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com> Co-authored-by: bofeng huang <bofenghuang7@gmail.com> Co-authored-by: Sylvain Gugger <Sylvain.gugger@gmail.com>	2023-02-01 19:09:36 +00:00
Quentin Lhoest	074d6b75fd	Simplify column_names in run_clm/mlm (#21382 ) * simplify column_names in run_clm * simplify column_names in run_mlm * minor	2023-01-31 15:23:47 +01:00
Stas Bekman	98d88b23f5	[`run_(clm\|mlm).py` examples] add streaming dataset support (#21343 ) * [run_clm example] add streaming dataset support * unrefactor kwargs * fix * fix * require datasets>=2.0.0 * port to mlm	2023-01-30 14:01:35 -08:00
dependabot[bot]	36b668fa06	Bump onnx from 1.11.0 to 1.13.0 in /examples/research_projects/decision_transformer (#21331 ) Bump onnx in /examples/research_projects/decision_transformer Bumps [onnx](https://github.com/onnx/onnx) from 1.11.0 to 1.13.0. - [Release notes](https://github.com/onnx/onnx/releases) - [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md) - [Commits](https://github.com/onnx/onnx/compare/v1.11.0...v1.13.0) --- updated-dependencies: - dependency-name: onnx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-27 10:13:13 -05:00
Matt	071529bd54	Use return_tensors="np" instead of "tf" (#21266 ) Return NP instead of TF tensors for our data loading pipeline	2023-01-24 13:37:49 +00:00
Sylvain Gugger	7119bb052a	v4.27.0.dev0	2023-01-23 16:52:35 -05:00
Mostafa Elhoushi	5603f78fc4	Add scikit-learn dependency to train langage-modeling (#21229 )	2023-01-23 09:54:45 -05:00
amyeroberts	4bc18e7a83	Update examples with image processors (#21155 ) * Update examples to use image processors * Small fixes * Resolve conflicts	2023-01-19 15:14:58 +00:00
dependabot[bot]	f70ee51029	Bump future from 0.18.2 to 0.18.3 in /examples/research_projects/visual_bert (#21173 ) Bump future in /examples/research_projects/visual_bert Bumps [future](https://github.com/PythonCharmers/python-future) from 0.18.2 to 0.18.3. - [Release notes](https://github.com/PythonCharmers/python-future/releases) - [Changelog](https://github.com/PythonCharmers/python-future/blob/master/docs/changelog.rst) - [Commits](https://github.com/PythonCharmers/python-future/compare/v0.18.2...v0.18.3) --- updated-dependencies: - dependency-name: future dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-18 11:17:35 -05:00
dependabot[bot]	0194665c33	Bump future from 0.18.2 to 0.18.3 in /examples/research_projects/lxmert (#21169 ) Bumps [future](https://github.com/PythonCharmers/python-future) from 0.18.2 to 0.18.3. - [Release notes](https://github.com/PythonCharmers/python-future/releases) - [Changelog](https://github.com/PythonCharmers/python-future/blob/master/docs/changelog.rst) - [Commits](https://github.com/PythonCharmers/python-future/compare/v0.18.2...v0.18.3) --- updated-dependencies: - dependency-name: future dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-18 11:16:43 -05:00
Sylvain Gugger	05e72aa0c4	Adapt repository creation to latest hf_hub (#21158 ) * Adapt repository creation to latest hf_hub * Update all examples * Fix other tests, add Flax examples * Address review comments	2023-01-18 11:14:00 -05:00
Observer46	ff8dcb5efa	Fix arguments passed to predict function in QA Seq2seq training script (#21026 ) fix args passed to predict function	2023-01-06 07:19:42 -05:00
Roy Hvaara	35a7052b61	[NumPy] Remove references to deprecated NumPy type aliases (#21022 ) [NumPy] Remove references to deprecated NumPy type aliases. This change replaces references to a number of deprecated NumPy type aliases (np.bool, np.int, np.float, np.complex, np.object, np.str) with their recommended replacement (bool, int, float, complex, object, str). NumPy 1.24 drops the deprecated aliases, so we must remove uses before updating NumPy. Co-authored-by: Peter Hawkins <phawkins@google.com> Co-authored-by: Peter Hawkins <phawkins@google.com>	2023-01-05 13:02:10 -05:00
Magnus Pierrau	1d21471c78	Added mask_time_prob and mask_time_length arguments to wav2vec2 pretraining script (#20985 ) Added mask_time_prob and mask_time_length arguments to wav2vec2 pretraining script and readme - new branch	2023-01-05 16:24:55 +00:00
dependabot[bot]	b7417bee87	Bump gitpython from 3.0.2 to 3.1.30 in /examples/research_projects/distillation (#21011 ) Bump gitpython in /examples/research_projects/distillation Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.0.2 to 3.1.30. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.0.2...3.1.30) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-04 15:36:42 -05:00
dependabot[bot]	05b736c16e	Bump gitpython from 3.1.18 to 3.1.30 in /examples/research_projects/decision_transformer (#21010 ) Bump gitpython in /examples/research_projects/decision_transformer Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.18 to 3.1.30. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.18...3.1.30) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-04 15:36:33 -05:00
milyiyo	3b309818e7	Refactor the function get_results (#20999 )	2023-01-04 12:05:36 -05:00
Wang, Yi	9c9fe89f84	[run_clm example] add torch_dtype option for model load. (#20971 ) * [run_clm example] add torch_dtype option for model load. for BLOOM 175B model. peak memory will reduce about 350G for inference. the weight of BLOOM in model hub is bfloat16 Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * add other type in option * fix style Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>	2023-01-03 09:33:11 -05:00
Alex Hedges	0b686a8a1e	Remove non-breaking spaces (#20929 ) * Remove non-breaking space in comment It was likely added unintionally. * Remove remaining non-breaking spaces	2022-12-29 02:12:40 -05:00
Márton Makrai	3830b3f74a	Fixes typo in the help text for --max_length (#20883 )	2022-12-24 02:07:06 -05:00
NielsRogge	d87e381f93	[Examples] Update big table (#20845 ) Update big table Co-authored-by: Niels Rogge <nielsrogge@Nielss-MBP.localdomain>	2022-12-21 11:34:31 +01:00
fzyzcjy	ae3cbbcaf6	Fix tiny typo (#20841 ) * Fix typo * Update README.md * Update run_mlm_flax_stream.py * Update README.md	2022-12-20 03:17:59 -05:00
dependabot[bot]	da95f6ca4c	Bump certifi in /examples/research_projects/visual_bert (#20673 ) Bumps [certifi](https://github.com/certifi/python-certifi) from 2020.6.20 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](https://github.com/certifi/python-certifi/compare/2020.06.20...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-12-08 11:15:42 -05:00
dependabot[bot]	efd7c021ee	Bump certifi in /examples/research_projects/decision_transformer (#20677 ) Bumps [certifi](https://github.com/certifi/python-certifi) from 2021.10.8 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](https://github.com/certifi/python-certifi/compare/2021.10.08...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-12-08 11:15:11 -05:00
dependabot[bot]	9e33e19bf5	Bump certifi in /examples/research_projects/lxmert (#20672 ) Bumps [certifi](https://github.com/certifi/python-certifi) from 2020.6.20 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](https://github.com/certifi/python-certifi/compare/2020.06.20...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-12-08 11:14:54 -05:00
Emmanuel Schmidbauer	0526a075c5	run_speech_recognition_seq2seq.py: add cache_dir param to dataset (#20540 )	2022-12-07 18:23:16 +00:00
Francisco Kurucz	f821bea0ad	Fix link to speech encoder decoder model in speech recognition readme (#20633 )	2022-12-06 15:46:41 -05:00
Wang, Yi	ae06bce888	exclude jit time from the speed metric calculation of evaluation and prediction (#20553 ) Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>	2022-12-06 07:37:01 -05:00
Sylvain Gugger	60d1f31bb0	v4.26.0.dev0	2022-12-01 16:19:33 -05:00
Wang, Yi	d752337baa	QnA example: add speed metric (#20522 )	2022-12-01 12:04:19 -05:00
Pi Esposito	fb2b45e562	add in layer gpt2 tokenizer (#20421 ) * add minimal working gpt2 tokenizer * graph mode and output equivalence tests working * not today tensorflow. serialization test passing! * fix style, documentation, docstrings and all that jazz * passing consistency checks * move keras nlp to tf dependencies * fix tf modeling utils and gpt2 attention to enable compiling * fix (I hope) keras nlp dependencies * rever changes on generation * remove debug prints * remove redundant tf dummy objects * add from config, get config and max length settings to address review * let flake ignore the error on distillation you are welcome * test from config * add padding test * address sgugger review	2022-11-29 10:02:40 -05:00
Katie Le	667ccea722	Replace assertion with ValueError exceptions in run_image_captioning_flax.py (#20365 ) * replace 4 asserts with ValueError exception for control flow * Update examples/flax/image-captioning/run_image_captioning_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * Update examples/flax/image-captioning/run_image_captioning_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * reformatted file * uninstalled trasformers and applied make style Co-authored-by: Bibi <Bibi@katies-mac.local> Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>	2022-11-28 15:06:25 +00:00
dependabot[bot]	624ae09f5c	Bump pillow in /examples/research_projects/decision_transformer (#20378 ) Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.0.1 to 9.3.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/9.0.1...9.3.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-11-22 08:15:42 -05:00
Zachary Mueller	9d1ef009b8	Fix flakey test with seed (#20318 )	2022-11-18 11:33:25 -05:00
Sylvain Gugger	a3f7458066	Pin to the right version...	2022-11-18 07:12:55 -05:00
Sanchit Gandhi	c29a2f7c9c	[ASR Examples] Update README for Whisper (#20230 ) * [ASR Examples] Update README for seq2seq * add language info * add training results * re-word	2022-11-18 11:24:25 +00:00
Zachary Mueller	441811ecd7	Fix summarization script (#20286 )	2022-11-16 15:57:07 -05:00
Jiahao Li	9681f052a1	Fix result saving errors of pytorch examples (#20276 )	2022-11-16 09:51:04 -05:00
Zachary Mueller	822ae69c1b	Update reqs to include min gather_for_metrics Accelerate version (#20242 ) * Update reqs to include min gather_for_metrics Accelerate version * Other reqs	2022-11-15 13:28:00 -05:00
Muhammad Sakib Khan Inan	777b1bfe62	New logging support to "Trainer" Class (ClearML Logger) (#20184 ) * Init Update * ClearML Callbacks integration * update corrections * args reporting updated * {'tensorboard': False, 'pytorch': False} * ClearML Tests added * add clearml * output_uri=True in Task.init * reformatted integrations.py * reformatted and fixed * IF-ELSE statement issue on "has_clearml" resolved * Add clearml in main callback docs * Add additional clearml documentation * Update src/transformers/integrations.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Accept suggestion Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Accept suggestion Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Small change in comments * Make style clearml * Accept suggestion Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: Victor Sonck <victor.sonck@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2022-11-15 10:08:59 -05:00
Yih-Dar	cf7b98b807	Fix `run_clip.py` (#20234 ) Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2022-11-15 15:45:21 +01:00
Ming Liu	36b063ed4f	Update README.md (#20188 ) There is typo in the original hyperlink. Below is the original version: Based on the script [`run_translation_no_trainer.py`](https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translationn_no_trainer.py).	2022-11-14 12:53:02 -05:00
Sanchit Gandhi	af1a7c8ca3	[Examples] Generalise Seq2Seq ASR to handle Whisper (#19519 ) * merge conflicts * bos and eos in datacollator * (temp) hardcode removal of attention mask * freeze encoder * actually freeze encoder * set max length / num beams according to gen kwargs * (temp) fix tests * don't pop attn mask * override return attention mask config from Hub * Hub configs updated 🤗 * final fixes * update type annotations * backward comp	2022-11-14 17:45:46 +00:00
Joao Gante	f270b960d6	Generate: move generation_.py src files into generation/.py (#20096 ) * move generation_.py src files into generation/.py * populate generation.__init__ with lazy loading * move imports and references from generation.xxx.object to generation.object	2022-11-09 15:34:08 +00:00
Tom Aarsen	b8112eddec	Replace unsupported facebookresearch/bitsandbytes (#20093 ) With https://github.com/TimDettmers/bitsandbytes, which is by the same author and is still being updated	2022-11-07 08:52:03 -05:00
bhuang	3502c202f9	Update README.md (#20063 )	2022-11-04 08:56:54 -04:00
Sylvain Gugger	06886d5a68	Only resize embeddings when necessary (#20043 ) * Only resize embeddings when necessary * Add comment	2022-11-03 12:05:04 -04:00
amyeroberts	a6b7759880	Add Image Processors (#19796 ) * Add CLIP image processor * Crop size as dict too * Update warning * Actually use logger this time * Normalize doesn't change dtype of input * Add perceiver image processor * Tidy up * Add DPT image processor * Add Vilt image processor * Tidy up * Add poolformer image processor * Tidy up * Add LayoutLM v2 and v3 imsge processors * Tidy up * Add Flava image processor * Tidy up * Add deit image processor * Tidy up * Add ConvNext image processor * Tidy up * Add levit image processor * Add segformer image processor * Add in post processing * Fix up * Add ImageGPT image processor * Fixup * Add mobilevit image processor * Tidy up * Add postprocessing * Fixup * Add VideoMAE image processor * Tidy up * Add ImageGPT image processor * Fixup * Add ViT image processor * Tidy up * Add beit image processor * Add mobilevit image processor * Tidy up * Add postprocessing * Fixup * Fix up * Fix flava and remove tree module * Fix image classification pipeline failing tests * Update feature extractor in trainer scripts * Update pad_if_smaller to accept tuple and int size * Update for image segmentation pipeline * Update src/transformers/models/perceiver/image_processing_perceiver.py Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com> * Update src/transformers/image_processing_utils.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/beit/image_processing_beit.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * PR comments - docstrings; remove accidentally added resize; var names * Update docstrings * Add exception if size is not in the right format * Fix exception check * Fix up * Use shortest_edge in tuple in script Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com> Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>	2022-11-02 11:57:36 +00:00
Sylvain Gugger	c3a93d8d82	v4.25.0.dev0	2022-10-31 21:48:40 -04:00
Sanchit Gandhi	f38a145418	[ASR] Update 'tasks' for model card (#19986 )	2022-10-31 16:50:17 +00:00
regisss	5d2d51a0fb	Fix LR (#19875 )	2022-10-26 08:35:53 -04:00
GMFTBY	71786b10c5	Adding the state-of-the-art contrastive search decoding methods for the codebase of generation_utils.py (#19477 ) * add: the contrastive search for generaton_utils * add: testing scripts for contrastive search under examples/text-generation * update the quality of codes * revise the docstring; make the generation_contrastive_search.py scripts; * revise the examples/pytorch/text-generation/run_generation_contrastive_search.py to the auto-APIs format * revise the necessary documents * fix: revise the docstring of generation_contrastive_search.py * Fix the code indentation * fix: revise the nits and examples in contrastive_search docstring. * fix the copyright * delete generation_contrastive_search.py * revise the logic in contrastive_search * update the intergration test and the docstring * run the tests over * add the slow decorate to the contrastive_search intergrate test * add more test * do the style, quality, consistency checks	2022-10-19 10:17:46 +01:00
amyeroberts	31ec424b3d	Add decorator to flaky test (#19674 )	2022-10-18 18:51:37 +01:00
David Yang	a23819ed6a	Clean up deprecation warnings (#19654 ) * Clean up deprecation warnings Notes: Changed some strings in tests to raw strings, which will change the literal content of the strings as they are fed into whatever machine handles them. Test cases for past in the past/past_key_values switch changed/removed due to warning of impending removal * Add PILImageResampling abstraction for PIL.Image.Resampling	2022-10-18 13:34:47 -04:00
Yifan Yang	94d7c3ba44	[Examples] make default preprocessing_num_workers=1 (#19684 ) * [Examples] make default preprocessing_num_workers=1 * [Examples] revert changes in research projects	2022-10-17 14:17:01 -04:00
Spacefish	2af36f957f	Add pillow to layoutlmv3 example requirements.txt (#19663 )	2022-10-17 08:41:57 -04:00
Duong A. Nguyen	4212bb0d60	[Re-submit] Compute true loss Flax examples (#19504 ) * Compute true loss * fixup * final * final * final * Update examples/flax/language-modeling/run_bart_dlm_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * jax.tree_map => jax.tree_util.tree_map * Compute true loss * final * fixup * final * final * Update examples/flax/language-modeling/run_bart_dlm_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * jax.tree_map => jax.tree_util.tree_map Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>	2022-10-13 11:33:36 +01:00
Sanchit Gandhi	eefcecaa35	[Examples] Fix typos in run speech recognition seq2seq (#19514 )	2022-10-12 15:33:22 +01:00
FilipposVentirozos	4ed0fa3676	Fix pytorch seq2seq qa (#19258 ) * fixed typo for SQuAD * Fixed the preprocess_validation_function function for the labels to reflect the remaining truncated instances * Rolled back the trainer_seq2seq_qa.py for UnboundLocalError: local variable 'metrics' referenced before assignment Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2022-10-12 08:33:44 -04:00
regisss	bb2cfd1824	Add multi-node conditions in trainer_qa.py and trainer_seq2seq.py (#19502 ) * Add multi-node conditions in trainer_qa.py and trainer_seq2seq.py * Code improvement	2022-10-11 22:48:56 -04:00
Lysandre	10100979ed	Dev version	2022-10-10 17:25:40 -04:00
Kaiyu Yang	e150c4e2fe	Fix the error message in run_t5_mlm_flax.py (#19282 )	2022-10-10 14:51:11 +01:00
wei zhao	7d5ce6802e	Fix typo in image-classification/README.md (#19424 ) Fix link typo of the following content. PyTorch version, Trainer PyTorch version, no Trainer	2022-10-10 09:16:58 -04:00
ddobokki	fa4bcd5274	edit: cast attention_mask to long in DataCollatorCTCWithPadding (#19369 ) * edit: casting attention_mask to long in DataCollatorCTCWithPadding * edit: casting attention_mask to long in DataCollatorCTCWithPadding	2022-10-07 10:05:48 -04:00
Ilaygoldman	ce2620194b	Change link of repojacking vulnerable link (#19393 ) The link to https://github.com/vasudevgupta7/bigbird is vulnerable to repojacking (it redirects to the orignial project that changed name), you should change the link to the current name of the project. if you won't change the link, an attacker can open the linked repository and attacks users that trust your links	2022-10-06 23:06:39 +02:00
Zachary Mueller	ad98642a82	Fix gather for metrics (#19360 )	2022-10-05 14:52:01 -04:00
dependabot[bot]	4c962d5e79	Bump joblib in /examples/research_projects/visual_bert (#19269 ) Bumps [joblib](https://github.com/joblib/joblib) from 0.16.0 to 1.2.0. - [Release notes](https://github.com/joblib/joblib/releases) - [Changelog](https://github.com/joblib/joblib/blob/master/CHANGES.rst) - [Commits](https://github.com/joblib/joblib/compare/0.16.0...1.2.0) --- updated-dependencies: - dependency-name: joblib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-03 23:57:50 +02:00
dependabot[bot]	c7ec0afce0	Bump joblib in /examples/research_projects/decision_transformer (#19270 ) Bumps [joblib](https://github.com/joblib/joblib) from 1.1.0 to 1.2.0. - [Release notes](https://github.com/joblib/joblib/releases) - [Changelog](https://github.com/joblib/joblib/blob/master/CHANGES.rst) - [Commits](https://github.com/joblib/joblib/compare/1.1.0...1.2.0) --- updated-dependencies: - dependency-name: joblib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-03 23:57:37 +02:00
dependabot[bot]	ca26277e33	Bump joblib from 0.16.0 to 1.2.0 in /examples/research_projects/lxmert (#19268 ) Bumps [joblib](https://github.com/joblib/joblib) from 0.16.0 to 1.2.0. - [Release notes](https://github.com/joblib/joblib/releases) - [Changelog](https://github.com/joblib/joblib/blob/master/CHANGES.rst) - [Commits](https://github.com/joblib/joblib/compare/0.16.0...1.2.0) --- updated-dependencies: - dependency-name: joblib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-03 23:49:35 +02:00
Divyanshu Kumar	c28d04e9e2	Update no_trainer script for summarization (#19277 ) * Update no_trainer script for summarization * removed unnecessary import * fixes notation mistake * removed: unused variable	2022-10-03 09:21:51 -04:00
Sylvain Gugger	0fc68a7e14	Fix seq2seq QA example	2022-09-28 15:45:49 -04:00
Tatsuki Okada	4a0b958d61	Fix trainer seq2seq qa.py evaluate log and ft script (#19208 ) * fix args option * fix trainer eval log * fix out of memory qa script * do isort, black, flake * fix tokenize target * take it back. * fix: comment	2022-09-28 10:55:46 -04:00
Arijit Mukherjee	ea540a5977	add wav2vec2_alignment (#16782 ) * add wav2vec2_alignment * Update alignment.py * Update examples/research_projects/wav2vec2/alignment.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Update examples/research_projects/wav2vec2/alignment.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Update examples/research_projects/wav2vec2/alignment.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Update examples/research_projects/wav2vec2/alignment.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Update README.md * fix style * fix imports * fix multithread * fix bash script * [@anton-l] Style fixes and docstrings * [@anton-l] Style fixes and docstrings * Update alignment.py fix blank id in backtrack Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: anton-l <aglozhkov@gmail.com>	2022-09-27 13:12:56 +02:00
Sylvain Gugger	c20b2c7e18	Use repo_type instead of deprecated datasets repo IDs (#19202 ) * Use repo_type instead of deprecated datasets repo IDs * Add missing one in doc	2022-09-26 09:50:48 -04:00
dependabot[bot]	98af4f9b54	Bump protobuf in /examples/research_projects/decision_transformer (#19176 ) Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.19.4 to 3.19.5. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.19.4...v3.19.5) --- updated-dependencies: - dependency-name: protobuf dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-09-26 14:55:16 +02:00
Enze	5da6afdd8d	Update run_clip.py (#19130 ) The overwrite_cache parameter is declared twice.	2022-09-23 20:48:41 +02:00
dependabot[bot]	74a3ea4737	Bump oauthlib in /examples/research_projects/decision_transformer (#19080 ) Bumps [oauthlib](https://github.com/oauthlib/oauthlib) from 3.2.0 to 3.2.1. - [Release notes](https://github.com/oauthlib/oauthlib/releases) - [Changelog](https://github.com/oauthlib/oauthlib/blob/master/CHANGELOG.rst) - [Commits](https://github.com/oauthlib/oauthlib/compare/v3.2.0...v3.2.1) --- updated-dependencies: - dependency-name: oauthlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-09-22 17:01:40 +02:00
Matt	83dc6377d0	Reduce LR for TF MLM example test (#19156 )	2022-09-22 08:51:27 -04:00
Leandro von Werra	ef6741fe65	Fix GLUE MNLI when using `max_eval_samples` (#18722 )	2022-09-21 09:33:22 +02:00
Santiago Castro	06f341de4f	Add a missing space in a script arg documentation (#19113 )	2022-09-20 21:43:32 +02:00
Ryan X	36e356caa4	Fix: update ltp word segmentation call in mlm_wwm (#19047 ) * Fix: update ltp word segmentation call in mlm_wwm * Fix: update ltp word segmentation call in mlm_wwm * Fix: update ltp word segmentation call in mlm_wwm	2022-09-20 09:20:38 -04:00
dependabot[bot]	7d0486c106	Bump mako in /examples/research_projects/decision_transformer (#19077 ) Bumps [mako](https://github.com/sqlalchemy/mako) from 1.2.0 to 1.2.2. - [Release notes](https://github.com/sqlalchemy/mako/releases) - [Changelog](https://github.com/sqlalchemy/mako/blob/main/CHANGES) - [Commits](https://github.com/sqlalchemy/mako/commits) --- updated-dependencies: - dependency-name: mako dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-09-16 22:15:02 +02:00
Lysandre	16913b3c92	Dev version	2022-09-14 14:58:20 -04:00
Rahul A R	00fc9217d1	Fixed bug which caused overwrite_cache to always be True (#19000 ) * fixed bug which caused overwrite_cache to always be True (#18967). * reformatting changes	2022-09-13 11:29:48 -04:00
Rafał Jankowski	85125fcffd	Neptune.ai integration improvements (#18934 ) * NeptuneCallback improvements * After review suggestions and deduplication of initial run * Added volatile checkpoints support due to missing post-rebase commit * Update README per review comments - Remove list formatting - Correct Neptune docs link Co-authored-by: Sabine <sabine.nyholm@neptune.ai>	2022-09-09 11:37:34 -04:00
Sanchit Gandhi	e6f221c8d4	[JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* (#18361 ) * [JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* * fix double tree_util	2022-09-09 15:18:56 +02:00
Nicholas Broad	4f299b2446	Accelerator end training (#18910 ) * add accelerator.end_training() Some trackers need this to end their runs. * fixup and quality * add space * add space again ?!?	2022-09-07 07:46:26 -04:00
arun99481	3b19c0317b	updating gather function with gather_for_metrics in run_wav2vec2_pretraining (#18877 ) Co-authored-by: Arun Rajaram <arunrajaram@Aruns-MacBook-Pro.local>	2022-09-06 07:36:37 -04:00
Sylvain Gugger	c61f116b63	Tie weights after preparing the model in run_clm (#18855 )	2022-09-01 12:06:56 -04:00
Rahul A R	e9442440fc	streamlining 'checkpointing_steps' parsing (#18755 )	2022-08-25 11:00:38 -04:00
Rahul A R	c55d6e4e10	examples/run_summarization_no_trainer: fixed incorrect param to hasattr (#18720 ) * fixed incorrect param to hasattr * simplified condition checks * code cleanup	2022-08-24 12:12:42 -04:00
dependabot[bot]	e49c71fc4c	Bump nbconvert from 6.3.0 to 6.5.1 in /examples/research_projects/lxmert (#18742 ) Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.3.0 to 6.5.1. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](https://github.com/jupyter/nbconvert/compare/6.3.0...6.5.1) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-24 06:12:56 -04:00
dependabot[bot]	5b24949669	Bump nbconvert in /examples/research_projects/visual_bert (#18741 ) Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.3.0 to 6.5.1. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](https://github.com/jupyter/nbconvert/compare/6.3.0...6.5.1) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-24 06:12:48 -04:00
Atharva Ingle	d90a36d192	remove check for main process for trackers initialization (#18706 )	2022-08-22 11:16:27 -04:00
Atharva Ingle	e54a1b49aa	`model.tie_weights()` should be applied after `accelerator.prepare()` (#18676 ) * `model.tie_weights()` should be applied after `accelerator.prepare` Weight tying should be done after the model has been moved to XLA device as mentioned on PyTorch/XLA Troubleshooting guide [here](https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md#xla-tensor-quirks) * format code	2022-08-18 13:46:57 -04:00
Loubna Ben Allal	bbbb453e58	Add an examples folder for code downstream tasks (#18679 ) * add examples subfolder * mention examples in codeparrot readme * use Trainer optimizer and scheduler type and add output_dir as argument * add example of text-to-python and python-to-text models * mention the downstream examples in the readme * fix typo	2022-08-18 18:24:24 +02:00
Zachary Mueller	358fc18613	Add evaluate to examples requirements (#18666 )	2022-08-18 10:57:39 -04:00
Stefan Schweter	358478e729	Examples: add Bloom support for token classification (#18632 ) * examples: add Bloom support for token classification (FLAX, PyTorch and TensorFlow) * examples: remove support for Bloom in token classication (FLAX and TensorFlow currently have no support for it)	2022-08-17 09:50:57 +02:00
zhoutang776	25e651a2de	Update run_translation_no_trainer.py (#18637 ) * Update run_translation_no_trainer.py found an error in selecting `no_decay` parameters and some small modifications when the user continues to train from a checkpoint * fixs `no_decay` and `resume_step` issue 1. change `no_decay` list 2. if use continue to train their model from provided checkpoint, the `resume_step` will not be initialized properly if `args.gradient_accumulation_steps != 1`	2022-08-16 13:25:57 -04:00
Karim Foda	d6eeb87170	Flax Remat for LongT5 (#17994 ) * [Flax] Add remat (gradient checkpointing) * fix variable naming in test * flip: checkpoint using a method * fix naming * fix class naming * apply PVP's suggestions from code review * add gradient_checkpointing to examples * Add gradient_checkpointing to run_mlm_flax * Add remat to longt5 * Add gradient checkpointing test longt5 * Fix args errors * Fix remaining tests * Make fixup & quality fixes * replace kwargs * remove unecessary kwargs * Make fixup changes * revert long_t5_flax changes * Remove return_dict and copy to LongT5 * Remove test_gradient_checkpointing Co-authored-by: sanchit-gandhi <sanchit@huggingface.co>	2022-08-14 16:27:13 +01:00
dependabot[bot]	05d3a43c59	Bump nbconvert in /examples/research_projects/visual_bert (#18566 ) Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.0.1 to 6.3.0. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](https://github.com/jupyter/nbconvert/compare/6.0.1...6.3.0) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-11 10:47:31 -04:00
dependabot[bot]	713ab6fde5	Bump nbconvert from 6.0.1 to 6.3.0 in /examples/research_projects/lxmert (#18565 ) Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.0.1 to 6.3.0. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](https://github.com/jupyter/nbconvert/compare/6.0.1...6.3.0) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-11 10:47:19 -04:00
Matt	6eb51450fa	TF Examples Rewrite (#18451 ) * Finished QA example * Dodge a merge conflict * Update text classification and LM examples * Update NER example * New Keras metrics WIP, fix NER example * Update NER example * Update MC, summarization and translation examples * Add XLA warnings when shapes are variable * Make sure batch_size is consistently scaled by num_replicas * Add PushToHubCallback to all models * Add docs links for KerasMetricCallback * Add docs links for prepare_tf_dataset and jit_compile * Correct inferred model names * Don't assume the dataset has 'lang' * Don't assume the dataset has 'lang' * Write metrics in text classification * Add 'framework' to TrainingArguments and TFTrainingArguments * Export metrics in all examples and add tests * Fix training args for Flax * Update command line args for translation test * make fixup * Fix accidentally running other tests in fp16 * Remove do_train/do_eval from run_clm.py * Remove do_train/do_eval from run_mlm.py * Add tensorflow tests to circleci * Fix circleci * Update examples/tensorflow/language-modeling/run_mlm.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/test_tensorflow_examples.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/translation/run_translation.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/token-classification/run_ner.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Fix save path for tests * Fix some model card kwargs * Explain the magical -1000 * Actually enable tests this time * Skip text classification PR until we fix shape inference * make fixup Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>	2022-08-10 16:49:51 +01:00
Rasmus Arpe Fogh Jensen	a765b68aa6	Update no_trainer.py scripts to include accelerate gradient accumulation wrapper (#18473 ) * Added accelerate gradient accumulation wrapper to run_image_classification_no_trainer.py example script * make fixup changes * PR comments * changed input to Acceletor based on PR comment, ran make fixup * Added comment explaining the sync_gradients statement * Fixed lr scheduler max steps * Changed run_clm_no_trainer.py script to use accelerate gradient accum wrapper * Fixed all scripts except wav2vec2 pretraining to use accelerate gradient accum wrapper * Added accelerate gradient accum wrapper for wav2vec2_pretraining_no_trainer.py script * make fixup and lr_scheduler step inserted back into run_qa_beam_search_no_trainer.py * removed changes to run_wav2vec2_pretraining_no_trainer.py script and fixed using wrong constant in qa_beam_search_no_trainer.py script	2022-08-08 15:52:47 -04:00
Sylvain Gugger	70b0d4e193	Fix compatibility with 1.12 (#17925 ) * Fix compatibility with 1.12 * Remove pin from examples requirements * Update torch scatter version * Fix compatibility with 1.12 * Remove pin from examples requirements * Update torch scatter version * fix torch.onnx.symbolic_opset12 import * Reject bad version Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2022-08-08 09:53:08 -04:00
regisss	88a0ce57bb	Add seed setting to image classification example (#18519 )	2022-08-08 08:08:11 -04:00
Julien Chaumond	9129fd0377	`transformers-cli login` => `huggingface-cli login` (#18490 ) * zero chance anyone's using that constant no? * `transformers-cli login` => `huggingface-cli login` * `transformers-cli repo create` => `huggingface-cli repo create` * `make style`	2022-08-06 09:42:55 +02:00
Julien Chaumond	8d1f9039d0	Just re-reading the whole doc every couple of months 😬 (#18489 ) * Delete valohai.yaml * NLP => ML * typo * website supports https * datasets * 60k + modalities * unrelated link fixing for accelerate * Ok those links were actually broken * Fix link * Make `AutoTokenizer` auto-link * wording tweak * add at least one non-nlp task	2022-08-06 09:38:55 +02:00
Kian Sierra McGettigan	0bf1e1aca4	Update no trainer examples for QA and Semantic Segmentation (#18474 ) * swag_no_trainer updated for with gather_metrics * Removed unused variable samples_seen * updated examples with gather_for_metrics	2022-08-04 13:22:19 -04:00
Kian Sierra McGettigan	330247ede2	Update no trainer scripts for multiple-choice (#18468 ) * swag_no_trainer updated for with gather_metrics * Removed unused variable samples_seen	2022-08-04 07:29:32 -04:00
LSinev	02b176c4ce	Fix torch version comparisons (#18460 ) Comparisons like version.parse(torch.__version__) > version.parse("1.6") are True for torch==1.6.0+cu101 or torch==1.6.0+cpu version.parse(version.parse(torch.__version__).base_version) are preferred (and available in pytorch_utils.py	2022-08-03 13:37:18 -04:00
Ritik Nandwal	3db4378bd7	Update no trainer scripts for language modeling and image classification examples (#18443 ) * Update no_trainer script for image-classification * Update no_trainer scripts for language-modeling examples * Remove unused variable * Removing truncation from losses array for language modeling examples	2022-08-03 08:33:18 -04:00
Yih-Dar	5546fb61ab	fix run_clip README (#18332 ) Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2022-08-02 19:14:46 +02:00
Duong A. Nguyen	3909d7f139	Add Flax BART pretraining script (#18297 ) * add bart pretraining flax script * fixup * add bart pretraining flax script * add BART to README * add BART to README * add BART to README * add BART to README * add BART to README * add bos eos document * Update README.md * Update README.md * Update examples/flax/language-modeling/run_bart_dlm_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * final * final * final * remove use_auth_token ing from_config Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>	2022-08-01 12:06:30 -04:00
Sylvain Gugger	941d233153	Fix ROUGE add example check and update README (#18398 ) * Fix ROUGE add example check and update README * Stay consistent in values	2022-08-01 11:14:49 -04:00
Ogundepo Odunayo	679d68a11b	Correct the spelling of bleu metric (#18375 )	2022-08-01 07:51:27 -04:00
atturaioe	1f84399171	Migrate metric to Evaluate in Pytorch examples (#18369 ) * Migrate metric to Evaluate in pytorch examples * Remove unused imports	2022-08-01 07:40:25 -04:00
dependabot[bot]	25ec12eaf7	Bump mistune from 0.8.4 to 2.0.3 in /examples/research_projects/lxmert (#18370 ) Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3. - [Release notes](https://github.com/lepture/mistune/releases) - [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst) - [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3) --- updated-dependencies: - dependency-name: mistune dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-01 04:46:57 -04:00
dependabot[bot]	a7360385f4	Bump mistune in /examples/research_projects/visual_bert (#18371 ) Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3. - [Release notes](https://github.com/lepture/mistune/releases) - [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst) - [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3) --- updated-dependencies: - dependency-name: mistune dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-01 04:46:31 -04:00
Sylvain Gugger	986526a0e4	Replace `as_target` context managers by direct calls (#18325 ) * Preliminary work on tokenizers * Quality + fix tests * Treat processors * Fix pad * Remove all uses of in tests, docs and examples * Replace all as_target_tokenizer * Fix tests * Fix quality * Update examples/flax/image-captioning/run_image_captioning_flax.py Co-authored-by: amyeroberts <amy@huggingface.co> * Style Co-authored-by: amyeroberts <amy@huggingface.co>	2022-07-29 08:09:09 -04:00
Vijay S Kalmath	da503ea02f	Migrate metrics used in flax examples to Evaluate (#18348 ) Currently, tensorflow examples use the `load_metric` function from Datasets library, commit migrates function call to `load` function from Evaluate library.	2022-07-28 15:06:23 -04:00
Vijay S Kalmath	a2586795e5	Migrate metric to Evaluate library for tensorflow examples (#18327 ) * Migrate metric to Evaluate library in tf examples Currently tensorflow examples use `load_metric` function from Datasets library , commit migrates function call to `load` function to Evaluate library. Fix for #18306 * Migrate metric to Evaluate library in tf examples Currently tensorflow examples use `load_metric` function from Datasets library , commit migrates function call to `load` function to Evaluate library. Fix for #18306 * Migrate `metric` to Evaluate for all tf examples Currently tensorflow examples use `load_metric` function from Datasets library , commit migrates function call to `load` function to Evaluate library.	2022-07-28 14:24:27 -04:00
Loubna Ben Allal	286a18fa00	Fix codeparrot deduplication - ignore whitespaces (#18023 ) * ignore whitspaces for hash * reformat code * Update README.md	2022-07-28 15:58:26 +02:00
Lysandre	c89a592e87	Dev version	2022-07-27 17:13:57 +02:00
Sanchit Gandhi	7490a97cac	[Flax] Fix incomplete batches in example scripts (#17863 ) * [Flax] Fix incomplete batches in example scripts * fix dataloader batching * convert jnp batch idxs to np array * add missing `pad_shard_unpad` to final prediction generate step * only `pad_shard_unpad` at inference time * merge conflicts * remove incomplete batch step from eval * fix run_qa.py * add `pad_shard_unpad` to run_flax_ner.py * add `pad_shard_unpad` to run_flax_glue.py * add `pad_shard_unpad` to run_image_classification.py * make style * fix mlm flax eval batches * remove redundant imports	2022-07-27 15:50:47 +01:00
Sylvain Gugger	cf32b2ee42	Remove all uses of six (#18318 ) * Remove all uses of six * fix quality	2022-07-27 08:39:09 -04:00
Duong A. Nguyen	170fcaa604	Generalize decay_mask_fn to apply mask to all LayerNorm params (#18273 ) * generalize decay_mask_fn to find all layernorm params * fixup * generalising decay_mask_fn	2022-07-27 12:23:57 +01:00
Loubna Ben Allal	1d71ad8905	Update CodeParrot readme to include training in Megatron (#17798 ) * add info about megatron training * upload models and datasets from CodeParrot organization * upload models and datasets from CodeParrot organization * Update examples/research_projects/codeparrot/README.md Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * Update examples/research_projects/codeparrot/README.md Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * Update examples/research_projects/codeparrot/README.md Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * Update examples/research_projects/codeparrot/README.md Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * Update examples/research_projects/codeparrot/README.md Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * fix typo and add comment about codeparrot vs megatron Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>	2022-07-27 11:59:08 +02:00
Zachary Mueller	99eb9b523f	Fix `no_trainer` CI (#18242 ) * Fix all tests	2022-07-21 14:44:57 -04:00
Duong A. Nguyen	4bea6584e3	Remove use_auth_token from the from_config method (#18192 ) * remove use_auth_token from from_config * restore use_auth_token from_pretrained run_t5_mlm_flax	2022-07-19 08:13:20 +02:00
John Giorgi	a4f97e6ce0	Fix incorrect type hint for lang (#18161 )	2022-07-18 09:53:18 +02:00
John Giorgi	c46d39f390	Fix check for falsey inputs in run_summarization (#18155 )	2022-07-18 09:50:32 +02:00
John Giorgi	fde22c75a1	Add summarization name mapping for MultiNews (#18117 ) * Add summarization name mapping for MultiNews * Add summarization name mapping for MultiNews	2022-07-13 08:19:20 -04:00
Duong A. Nguyen	1e8140caad	Fix RESOURCE_EXHAUSTED error when dealing with large datasets in Flax example scripts (#18069 ) * Fix RESOURCE_EXHAUSTED error for large datasets on Flax example scripts * using np.permutation for creating batch_idx * train_samples_idx -> training_samples_idx * fix type hints	2022-07-11 15:59:08 +02:00
Yulv-git	95113d1365	Fix some typos. (#17560 ) * Fix some typos. Signed-off-by: Yulv-git <yulvchi@qq.com> * Fix typo. Signed-off-by: Yulv-git <yulvchi@qq.com> * make fixup.	2022-07-11 05:00:13 -04:00
ADAning	bf37e5c7f6	Fix T5 incorrect weight decay in Trainer and official summarization example (#18002 ) * Add ALL_LAYERNORM_LAYERS for LayerNorm * fix bug of appending layer norm	2022-07-06 09:44:19 -04:00
Zachary Mueller	7c4c6f6084	Fix all is_torch_tpu_available issues (#17936 ) * Fix all is_torch_tpu_available	2022-06-29 11:03:33 -04:00
Sylvain Gugger	5f1e67a566	Pin PyTorch in requirements as well	2022-06-28 15:56:10 -04:00
Zachary Mueller	75259b44bf	Properly calculate the total train iterations and recalculate num epochs in no_trainer scripts (#17856 )	2022-06-23 15:46:01 -04:00
Zachary Mueller	acb709d551	Change no trainer image_classification test (#17635 ) * Adjust test arguments and use a new example test	2022-06-23 11:11:16 -04:00
dependabot[bot]	c366ce1011	Bump numpy from 1.21.0 to 1.22.0 in /examples/research_projects/lxmert (#17817 ) Bumps [numpy](https://github.com/numpy/numpy) from 1.21.0 to 1.22.0. - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/HOWTO_RELEASE.rst) - [Commits](https://github.com/numpy/numpy/compare/v1.21.0...v1.22.0) --- updated-dependencies: - dependency-name: numpy dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-06-22 09:29:40 -04:00
dependabot[bot]	af0d21e741	Bump numpy in /examples/research_projects/visual_bert (#17816 ) Bumps [numpy](https://github.com/numpy/numpy) from 1.21.0 to 1.22.0. - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/HOWTO_RELEASE.rst) - [Commits](https://github.com/numpy/numpy/compare/v1.21.0...v1.22.0) --- updated-dependencies: - dependency-name: numpy dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-06-22 09:29:28 -04:00
Eran Hirsch	1357038164	Add logits_processor parameter, used by `generate`, to `Seq2SeqTrainer` methods `evaluate` and `predict` (#17805 ) * Add logits_processor parameter, used by `generate`, to `Seq2SeqTrainer` methods `evaluate` and `predict` * Add all generate parameters to `Seq2SeqTrainer`, and also to `QuestionAnsweringSeq2SeqTrainer` which overrides it * Remove `self._num_beams` from trainer classes * - Run fixup - Fix "Constraint" not exposed - Fix synced_gpus to actually read from param * Use kwargs * Copy kwargs before making changes to it * Fix style issues unused imports	2022-06-22 08:11:39 -04:00
Jia LI	da2bd2ae96	[CodeParrot] Near-deduplication with jaccard similarity (#17054 ) * deduplication draft * update style * update style test * dummy test main * rename modules * rename functions * return extremes in deduplicate_clusters * update style * cast str for gzip * update doc string * time processing * use dataset map to compute minhash * fill value for short token * remove da map method * update style * use share object to multiprocess * update style * use f-string and minor fix Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> Co-authored-by: Loubna Ben Allal <44069155+loubnabnl@users.noreply.github.com> * update style * use module parameters * change ds_dedup to ds_filter * save ds_dedup * mv test to script tests * make jaccard threshold a parameter of deduplicate_dataset * update style * add doc strings * update style * add doc string for DuplicationIndex * save files into data dir * update readme * Update examples/research_projects/codeparrot/README.md Co-authored-by: Loubna Ben Allal <44069155+loubnabnl@users.noreply.github.com> * make near deduplication optional * move near deduplication in README * Update examples/research_projects/codeparrot/README.md Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * use f string Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> Co-authored-by: Loubna Ben Allal <44069155+loubnabnl@users.noreply.github.com>	2022-06-21 14:23:36 +02:00
dependabot[bot]	e44a569fef	Bump notebook in /examples/research_projects/lxmert (#17743 ) Bumps [notebook](http://jupyter.org) from 6.4.10 to 6.4.12. --- updated-dependencies: - dependency-name: notebook dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-06-17 12:10:33 -04:00
dependabot[bot]	5089a2d412	Bump notebook in /examples/research_projects/visual_bert (#17742 ) Bumps [notebook](http://jupyter.org) from 6.4.10 to 6.4.12. --- updated-dependencies: - dependency-name: notebook dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-06-17 12:10:17 -04:00
Sylvain Gugger	7c6ec195ad	v4.21.0.dev0	2022-06-16 12:20:53 -04:00
Jeff Rasley	6ebeeeef81	Update requirements.txt (#17719 )	2022-06-15 13:51:41 -04:00
Shamane Siri	9068fa6c57	Rag end2end new (#17650 ) * check * update the RAG-end2end with new PL and RAY * removed unwanted comments	2022-06-14 14:56:32 +02:00
Loubna Ben Allal	3114df41f4	update README.md (#17657 ) - use CodeParrot scores of v1.1 - change evaluation command to use accelerate	2022-06-10 15:55:24 +02:00
Simon Brandeis	c99ddcc441	🐛 Properly raise `RepoNotFoundError` when not authenticated (#17651 ) * Raise RepoNotFoundError in case of 401 * Include changes from revert-17646-skip_repo_not_found * Add a comment * 💄 Code quality * 💚 Update `get_from_cache` test * 💚 Code quality & skip failing test	2022-06-10 15:41:53 +02:00
dependabot[bot]	1d463303fe	Bump cookiecutter in /examples/research_projects/decision_transformer (#17645 ) Bumps [cookiecutter](https://github.com/cookiecutter/cookiecutter) from 1.7.2 to 2.1.1. - [Release notes](https://github.com/cookiecutter/cookiecutter/releases) - [Changelog](https://github.com/cookiecutter/cookiecutter/blob/master/HISTORY.md) - [Commits](https://github.com/cookiecutter/cookiecutter/compare/1.7.2...2.1.1) --- updated-dependencies: - dependency-name: cookiecutter dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-06-10 04:27:51 -04:00
Sylvain Gugger	3cab90279f	Add examples telemetry (#17552 ) * Add examples telemetry * Alternative approach * Add to all other examples * Add to templates as well * Put framework separately * Same for TensorFlow	2022-06-07 11:57:52 -04:00
bhuang	254d9c068e	Update run_glue_no_trainer.py (#17546 )	2022-06-03 12:29:37 -04:00
Zachary Mueller	3766df4fe1	Fix flakey no-trainer test (#17515 )	2022-06-01 13:40:49 -04:00
fireindark707	028d4b7c8b	Deal with the error when task is regression (#16330 )	2022-06-01 11:15:53 -04:00
Sourab Mangrulkar	d156898f3b	Improve notrainer examples (#17449 ) * improve no-trainer examples * Trigger CI * adding comment to clarify tracker init on main process * Trigger CI * Trigger CI * Trigger CI	2022-05-28 00:06:31 +05:30
Patrick von Platen	a9eca74372	Wav2vec2 finetuning shared file system (#17423 ) * fix_torch_device_generate_test * remove @ * [Fix shared file system] Co-authored-by: Patrick von Platen <patrick@huggingface.co>	2022-05-25 22:04:43 +02:00
dependabot[bot]	1ef9a1ed4a	Bump tensorflow in /examples/research_projects/decision_transformer (#17400 ) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.0 to 2.8.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.0...v2.8.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-05-24 19:36:55 -04:00
NielsRogge	31ee80d556	Add LayoutLMv3 (#17060 ) * Make forward pass work * More improvements * Remove unused imports * Remove timm dependency * Improve loss calculation of token classifier * Fix most tests * Add docs * Add model integration test * Make all tests pass * Add LayoutLMv3FeatureExtractor * Improve integration test + make fixup * Add example script * Fix style * Add LayoutLMv3Processor * Fix style * Add option to add visual labels * Make more tokenizer tests pass * Fix more tests * Make more tests pass * Fix bug and improve docs * Fix import of processors * Improve docstrings * Fix toctree and improve docs * Fix auto tokenizer * Move tests to model folder * Move tests to model folder * change default behavior add_prefix_space * add prefix space for fast * add_prefix_spcae set to True for Fast * no space before `unique_no_split` token * add test to hightligh special treatment of added tokens * fix `test_batch_encode_dynamic_overflowing` by building a long enough example * fix `test_full_tokenizer` with add_prefix_token * Fix tokenizer integration test * Make the code more readable * Add tests for LayoutLMv3Processor * Fix style * Add model to README and update init * Apply suggestions from code review * Replace asserts by value errors * Add suggestion by @ducviet00 * Add model to doc tests * Simplify script * Improve README * a step ahead to fix * Update pair_input_test * Make all tokenizer tests pass - phew * Make style * Add LayoutLMv3 to CI job * Fix auto mapping * Fix CI job name * Make all processor tests pass * Make tests of LayoutLMv2 and LayoutXLM consistent * Add copied from statements to fast tokenizer * Add copied from statements to slow tokenizer * Remove add_visual_labels attribute * Fix tests * Add link to notebooks * Improve docs of LayoutLMv3Processor * Fix reference to section Co-authored-by: SaulLu <lucilesaul.com@gmail.com> Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>	2022-05-24 09:53:45 +02:00
Loubna Ben Allal	b48ac1a094	Fix CodeParrot training script (#17291 ) * average loss over batches and accumulated steps for tracking * fix layernorm weight decay * use AdamW from Pytorch instead of Transformers * add shuffling of sequences inside the batches * add shuffling of sequences inside the batches * add logging dir and reformat code * fix lr tracking * remove Mistral scaling * keep Mistral scaling * reformat code * fix error * fix error * use shuffling function from Pytorch * remove argument for shuffling batch sequences as it isn't optional * update package versions and install accelerate from source * remove unused package * Update loss average over accumulated steps Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * Update loss average over accumulated steps Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * use one shuffle buffer argument * compute avg_loss in one line Co-authored-by: Loubna ben allal <loubnabenallal@gmail.com> Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>	2022-05-23 12:55:35 +02:00
ddobokki	48c22691e3	Fix bug in Wav2Vec2 pretrain example (#17326 )	2022-05-19 22:42:44 +02:00
Zachary Mueller	1762ded30a	Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts (#17331 ) * Fix length in no_trainer examples * Add setup and teardown * Use new accelerator config generator to automatically make tests able to run based on environment	2022-05-18 14:17:40 -04:00
Sylvain Gugger	4710702837	Fix style	2022-05-18 10:46:40 -04:00
mraunak	5fdb54ece7	Add Information Gain Filtration algorithm (#16953 ) * Add information gain filtration algorithm * Complying with black requirements * Added author * Fixed import order * flake8 corrections Co-authored-by: Javier Turek <javier.turek@intel.com>	2022-05-18 10:39:02 -04:00
regisss	28a0811652	Improve mismatched sizes management when loading a pretrained model (#17257 ) - Add --ignore_mismatched_sizes argument to classification examples - Expand the error message when loading a model whose head dimensions are different from expected dimensions	2022-05-17 17:58:14 +02:00
Loubna Ben Allal	05a90579a8	CodeParrot data pretokenization (#16932 ) * add pretokenization arguments * add pretokenization script * add support for pretokenized data * reformat code * fix run command for training * fix model call from config * remove a package * add comments on pretokenization in the readme * remove explicit parallelization Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * update readme Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * update readme -remove username Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * update readme -remove username Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * keep data parallelization * reformat code * reformat code * update readme * reformat code * Update examples/research_projects/codeparrot/README.md Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> Co-authored-by: Loubna ben allal <loubnabenallal@gmail.com>	2022-05-16 15:32:16 +02:00
Loubna Ben Allal	e730e12567	Update codeparrot data preprocessing (#16944 ) * add new preprocessing arguments * add new filters * add new filters to readme * fix config and test count, update function names and docstrings * reformat code * update readme * Update readme * rename config_test filter Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * rename few_assignments filter Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * rename tokenizer in arguments Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> * rename functions and add limit_line argument for config_test filter * update threshold for config_test filter Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com> Co-authored-by: Loubna ben allal <loubnabenallal@gmail.com>	2022-05-16 14:43:25 +02:00
Kenneth Enevoldsen	71d18d0831	fixed bug in run_mlm_flax_stream.py (#17203 ) * fixed bug run_mlm_flax_stream.py Fixed bug caused by an update to tokenizer keys introduced in recent transformers versions (between `4.6.2` and `4.18.0`) where additional keys were introduced to the tokenizer output. * Update run_mlm_flax_stream.py * adding missing paranthesis * formatted to black * remove cols from dataset instead * reformat to black * moved rem. columns to map * formatted to black Co-authored-by: KennethEnevoldsen <kennethcenevolsen@gmail.com>	2022-05-16 13:40:27 +02:00
Sylvain Gugger	afe5d42d8d	Black preview (#17217 ) * Black preview * Fixup too! * Fix check copies * Use the same version as the CI * Bump black	2022-05-12 16:25:55 -04:00
Lysandre Debut	5294fa12ee	Dev version	2022-05-12 11:04:23 -04:00
Zachary Mueller	d719bcd46a	Fix all docs for accelerate install directions (#17145 )	2022-05-09 15:45:18 -04:00
Zachary Mueller	ef20390291	Update to build via git for accelerate (#17084 )	2022-05-04 09:42:36 -04:00
dependabot[bot]	2bf95e2b09	Bump notebook from 6.4.1 to 6.4.10 in /examples/research_projects/lxmert (#16634 ) Bumps [notebook](http://jupyter.org) from 6.4.1 to 6.4.10. --- updated-dependencies: - dependency-name: notebook dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-05-04 08:27:40 -04:00
dependabot[bot]	7a229ef446	Bump notebook in /examples/research_projects/visual_bert (#16635 ) Bumps [notebook](http://jupyter.org) from 6.4.1 to 6.4.10. --- updated-dependencies: - dependency-name: notebook dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-05-04 08:27:27 -04:00
Thomas Wang	db034660fb	Fix hashing for deduplication (#17048 )	2022-05-04 08:40:24 +02:00
Pavel Belevich	39f8eafc1b	Remove device parameter from create_extended_attention_mask_for_decoder (#16894 )	2022-05-03 11:06:11 -04:00
Zachary Mueller	f275e593bf	Fix no_trainer examples to properly calculate the number of samples (#17046 ) * Update all examples to properly calculate progress bar	2022-05-02 11:56:25 -04:00
Zachary Mueller	35d48db881	Update no_trainer examples to use new logger (#17044 ) * Propagate and fix imports	2022-05-02 11:56:15 -04:00
yujun	bdd690a74d	add torch.no_grad when in eval mode (#17020 ) * add torch.no_grad when in eval mode * make style quality	2022-05-02 07:49:19 -04:00
Zachary Mueller	3486a92a57	Fix savedir for by epoch (#16996 )	2022-04-28 13:49:45 -04:00
conan1024hao	1be8d56ec6	Add parameter --config_overrides for run_mlm_wwm.py (#16961 ) * dd parameter --config_overrides for run_mlm_wwm.py * linter	2022-04-28 10:44:55 -04:00
Zachary Mueller	60e1d883f1	Fixup no_trainer save logic (#16968 ) * Fixup all examples	2022-04-27 14:46:49 -04:00
Sylvain Gugger	c79bbc3ba5	Fix multiple deletions of the same files in save_pretrained (#16947 ) * Fix multiple deletions of the same files in save_pretrained * Add is_main_process argument	2022-04-27 12:28:42 -04:00
Leonid Boytsov	c82e017aa9	Misc. fixes for Pytorch QA examples: (#16958 ) 1. Fixes evaluation errors popping up when you train/eval on squad v2 (one was newly encountered and one that was previously reported Running SQuAD 1.0 sample command raises IndexError #15401 but not completely fixed). 2. Removes boolean arguments that don't use store_true. Please, don't use these: *ANY non-empty string is being converted to True in this case and this clearly is not the desired behavior (and it creates a LOT of confusion). 3. All no-trainer test scripts are now saving metric values in the same way (with the right prefix eval_), which is consistent with the trainer-based versions. 4. Adds forgotten model.eval() in the no-trainer versions. This improved some results, but not everything (see the discussion in the end). Please, see the F1 scores and the discussion below.	2022-04-27 08:51:39 -04:00
NielsRogge	479fdc4925	Add semantic script, trainer (#16834 ) * Add first draft * Improve script and README * Improve README * Apply suggestions from code review * Improve script, add link to resulting model * Add corresponding test * Adjust learning rate	2022-04-27 10:12:18 +02:00

... 4 5 6 7 8 ...

2548 Commits