transformers

mirror of https://github.com/huggingface/transformers.git synced 2025-07-05 22:00:09 +06:00

Author	SHA1	Message	Date
Yih-Dar	64bdbd888c	Don't set `finetuned_from` if it is a local path (#28482 ) * fix * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2024-01-15 11:38:20 +01:00
Joao Gante	4fb3d3a0f6	TF: purge `TFTrainer` (#28483 )	2024-01-12 16:56:34 +00:00
dependabot[bot]	c45ef1c0d1	Bump jinja2 from 2.11.3 to 3.1.3 in /examples/research_projects/decision_transformer (#28457 ) Bump jinja2 in /examples/research_projects/decision_transformer Bumps [jinja2](https://github.com/pallets/jinja) from 2.11.3 to 3.1.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/jinja/compare/2.11.3...3.1.3) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-01-12 15:28:55 +01:00
Alex Hedges	95091e1582	Set `cache_dir` for `evaluate.load()` in example scripts (#28422 ) While using `run_clm.py`,[^1] I noticed that some files were being added to my global cache, not the local cache. I set the `cache_dir` parameter for the one call to `evaluate.load()`, which partially solved the problem. I figured that while I was fixing the one script upstream, I might as well fix the problem in all other example scripts that I could. There are still some files being added to my global cache, but this appears to be a bug in `evaluate` itself. This commit at least moves some of the files into the local cache, which is better than before. To create this PR, I made the following regex-based transformation: `evaluate\.load$(.*?)$` -> `evaluate\.load$$1, cache_dir=model_args.cache_dir$`. After using that, I manually fixed all modified files with `ruff` serving as useful guidance. During the process, I removed one existing usage of the `cache_dir` parameter in a script that did not have a corresponding `--cache-dir` argument declared. [^1]: I specifically used `pytorch/language-modeling/run_clm.py` from v4.34.1 of the library. For the original code, see the following URL: `acc394c4f5/examples/pytorch/language-modeling/run_clm.py`.	2024-01-11 15:38:44 +01:00
dependabot[bot]	932ad8af7a	Bump fonttools from 4.31.1 to 4.43.0 in /examples/research_projects/decision_transformer (#28417 ) Bump fonttools in /examples/research_projects/decision_transformer Bumps [fonttools](https://github.com/fonttools/fonttools) from 4.31.1 to 4.43.0. - [Release notes](https://github.com/fonttools/fonttools/releases) - [Changelog](https://github.com/fonttools/fonttools/blob/main/NEWS.rst) - [Commits](https://github.com/fonttools/fonttools/compare/4.31.1...4.43.0) --- updated-dependencies: - dependency-name: fonttools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-01-10 11:22:43 +01:00
Lysandre	3ed3e3190c	Dev version	2023-12-13 18:29:31 +01:00
saswatmeher	a49f4acab3	Fix link in README.md of Image Captioning (#27969 ) Update the link for vision encoder decoder doc used by FlaxVisionEncoderDecoderModel link.	2023-12-12 08:07:15 -05:00
Adam Louly	4850aaba6f	fix no sequence length models error (#27522 ) * fix no sequence length models error * block size check --------- Co-authored-by: Adam Louly <adamlouly@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>	2023-12-11 18:01:26 +00:00
Phuc Van Phan	0410a29a2d	fix: fix gradient accumulate step for learning rate (#27667 )	2023-12-07 07:59:26 +01:00
Dave Berenbaum	fe41647afc	uses dvclive_test mode in examples/pytorch/test_accelerate_examples.py (#27763 )	2023-11-30 14:52:03 +01:00
Peter Pan	ce31508134	docs: replace torch.distributed.run by torchrun (#27528 ) * docs: replace torch.distributed.run by torchrun `transformers` now officially support pytorch >= 1.10. The entrypoint `torchrun`` is present from 1.10 onwards. Signed-off-by: Peter Pan <Peter.Pan@daocloud.io> * Update src/transformers/trainer.py with @ArthurZucker's suggestion Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> --------- Signed-off-by: Peter Pan <Peter.Pan@daocloud.io> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>	2023-11-27 16:26:33 +00:00
Mathias Nielsen	f31af3927f	[ examples] fix loading jsonl with load dataset in run translation example (#26924 ) * Renamed variable extension to builder_name * If builder name is jsonl change to json to align with load_datasets * Apply suggestions from code review Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com> --------- Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com>	2023-11-20 15:45:42 +01:00
V.Prasanna kumar	ffbcfc0166	Broken links fixed related to datasets docs (#27569 ) fixed the broken links belogs to dataset library of transformers	2023-11-17 13:44:09 -08:00
Arthur	651408a077	[`Styling`] stylify using ruff (#27144 ) * try to stylify using ruff * might need to remove these changes? * use ruf format andruff check * use isinstance instead of type comparision * use # fmt: skip * use # fmt: skip * nits * soem styling changes * update ci job * nits isinstance * more files update * nits * more nits * small nits * check and format * revert wrong changes * actually use formatter instead of checker * nits * well docbuilder is overwriting this commit * revert notebook changes * try to nuke docbuilder * style * fix feature exrtaction test * remve `indent-width = 4` * fixup * more nits * update the ruff version that we use * style * nuke docbuilder styling * leve the print for detected changes * nits * Remove file I/O Co-authored-by: charliermarsh <charlie.r.marsh@gmail.com> * style * nits * revert notebook changes * Add # fmt skip when possible * Add # fmt skip when possible * Fix * More ` # fmt: skip` usage * More ` # fmt: skip` usage * More ` # fmt: skip` usage * NIts * more fixes * fix tapas * Another way to skip * Recommended way * Fix two more fiels * Remove asynch Remove asynch --------- Co-authored-by: charliermarsh <charlie.r.marsh@gmail.com>	2023-11-16 17:43:19 +01:00
Lucain	fd65aa9818	Set `usedforsecurity=False` in hashlib methods (FIPS compliance) (#27483 ) * Set usedforsecurity=False in hashlib methods (FIPS compliance) * trigger ci * tokenizers version * deps * bump hfh version * let's try this	2023-11-16 14:29:53 +00:00
Matt	4989e73e2f	Update the TF pin for 2.15 (#27375 ) * Move the TF pin for 2.15 * make fixup	2023-11-16 13:47:43 +00:00
Phuc Van Phan	69c9b89fcb	docs: add docs for map, and add num procs to load_dataset (#27520 )	2023-11-16 13:16:19 +00:00
Matt	2e72bbab2c	Incorrect setting for num_beams in translation and summarization examples (#27519 ) * Remove the torch main_process_first context manager from TF examples * Correctly set num_beams=1 in our examples, and add a guard in GenerationConfig.validate() * Update src/transformers/generation/configuration_utils.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>	2023-11-15 18:18:54 +00:00
Adam Louly	e6522e49a7	Fixing the failure of models without max_position_embeddings attribute. (#27499 ) fix max pos issue Co-authored-by: Adam Louly <adamlouly@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>	2023-11-15 18:16:42 +00:00
Zach Mueller	a85ea4b19a	Fix wav2vec2 params (#27515 ) Fix test	2023-11-15 09:24:03 -05:00
Arthur	b97cab7e6d	Remove-auth-token (#27060 ) * don't use `use_auth_token`internally * let's use token everywhere * fixup	2023-11-13 14:20:54 +01:00
Yih-Dar	c8b6052ff6	Final fix of the accelerate installation issue (#27408 ) * fix * [test-all] commit * fix * [test-all] commit * [test-all] commit * fix * fix * fix * fix * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-11-09 18:52:29 +01:00
Dave Berenbaum	791ec370d1	Adds dvclive callback (#27352 ) * dvclive trainer callback * style fixes * dvclive link fixes	2023-11-09 12:19:31 +00:00
Zach Mueller	e9adb0c9cf	Change thresh in test (#27378 ) Change thresh	2023-11-09 04:44:36 -05:00
Zach Mueller	845aa832b7	Remove unused param from example script tests (#27354 ) Unused param	2023-11-08 09:07:32 -05:00
Zach Mueller	efa57cb234	Fix example tests from failing (#27353 ) * Fix example tests from failing * CHange thresh	2023-11-08 07:45:21 -05:00
Hz, Ji	b6dbfee0a2	moving example of benchmarking to legacy dir (#27337 ) move example of benchmarking to legacy	2023-11-08 09:27:37 +01:00
Lysandre	bc78fd1274	Dev version	2023-11-02 18:15:36 +01:00
Dong-geon Lee	25e6e9418c	Unify warning styles for better readability (#27184 )	2023-10-31 18:12:14 +00:00
Hz, Ji	cd19b19378	make tests of pytorch_example device agnostic (#27081 )	2023-10-30 14:56:41 +00:00
Gema Parreño	722e936491	[Typo fix] flag config in WANDB (#27130 ) typo fix flag config	2023-10-29 18:22:26 +00:00
Lucain	66b088faf0	Provide alternative when warning on use_auth_token (#27105 )	2023-10-27 14:32:54 +02:00
dependabot[bot]	3c2692407d	Bump urllib3 from 1.26.17 to 1.26.18 in /examples/research_projects/lxmert (#26888 ) Bump urllib3 in /examples/research_projects/lxmert Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.17...1.26.18) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-10-26 09:10:29 +02:00
dependabot[bot]	9c5240af14	Bump werkzeug from 2.2.3 to 3.0.1 in /examples/research_projects/decision_transformer (#27072 ) Bump werkzeug in /examples/research_projects/decision_transformer Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.2.3 to 3.0.1. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/2.2.3...3.0.1) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-10-26 08:56:28 +02:00
Michal Jamroz	e2d6d5ce57	Normalize only if needed (#26049 ) * Normalize only if needed * Update examples/pytorch/image-classification/run_image_classification.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * if else in one line * within block * one more place, sorry for mess * import order * Update examples/pytorch/image-classification/run_image_classification.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/pytorch/image-classification/run_image_classification_no_trainer.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>	2023-10-24 13:32:03 +01:00
YQ	f71c9ccf59	fix logit-to-multi-hot conversion in example (#26936 ) * fix logit to multi-hot converstion * add comments * typo	2023-10-23 12:33:05 +02:00
Matt	cbd278f0f6	Pin Keras for now (#26904 ) * Pin Keras for now out of paranoia * Add the keras pin to _tests_requirements.txt too * Make sure the Keras version matches the TF one * make fixup	2023-10-19 14:39:31 +01:00
dependabot[bot]	bece55d8f9	Bump urllib3 from 1.26.17 to 1.26.18 in /examples/research_projects/decision_transformer (#26889 ) Bump urllib3 in /examples/research_projects/decision_transformer Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.17...1.26.18) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-10-18 13:31:06 +02:00
dependabot[bot]	6d644d6852	Bump urllib3 from 1.26.17 to 1.26.18 in /examples/research_projects/visual_bert (#26890 ) Bump urllib3 in /examples/research_projects/visual_bert Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.17 to 1.26.18. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.17...1.26.18) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-10-18 04:30:50 -07:00
Tom Aarsen	40ea9ab2a1	Add many missing spaces in adjacent strings (#26751 ) Add missing spaces in adjacent strings	2023-10-12 10:28:40 +02:00
Zach Mueller	1d6a84749b	Fix checkpoint path in `no_trainer` scripts (#26733 ) checkpoint path	2023-10-11 16:16:27 +02:00
Roy Hvaara	fc63914399	[JAX] Replace uses of `jnp.array` in types with `jnp.ndarray`. (#26703 ) `jnp.array` is a function, not a type: https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.array.html so it never makes sense to use `jnp.array` in a type annotation. Presumably the intent was to write `jnp.ndarray` aka `jax.Array`. Co-authored-by: Peter Hawkins <phawkins@google.com>	2023-10-10 21:35:16 +02:00
jheitmann	3eceaa3637	Fix source_prefix default value (#26654 )	2023-10-10 20:49:10 +02:00
statelesshz	27597fea07	remove SharedDDP as it is deprecated (#25702 ) * remove SharedDDP as it was drepracated * apply review suggestion * make style * Oops,forgot to remove the compute_loss context manager in Seq2SeqTrainer. * remove the unnecessary conditional statement * keep the logic of IPEX * clean code * mix precision setup & make fixup --------- Co-authored-by: statelesshz <jihuazhong1@huawei.com>	2023-10-06 16:03:11 +02:00
Phuc Van Phan	6015f91a5a	refactor: change default block_size (#26229 ) * refactor: change default block_size * fix: return tf to origin * fix: change files to origin * rebase * rebase * rebase * rebase * rebase * rebase * rebase * rebase * refactor: add min block_size to files * reformat: add min block_size for run_clm tf	2023-10-04 15:31:38 +01:00
dependabot[bot]	fc296f419e	Bump pillow from 9.3.0 to 10.0.1 in /examples/research_projects/decision_transformer (#26580 ) Bump pillow in /examples/research_projects/decision_transformer Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.3.0 to 10.0.1. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/9.3.0...10.0.1) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-10-04 11:52:46 +02:00
Lysandre	bd6205919a	v4.35.0.dev0	2023-10-03 16:54:37 +02:00
dependabot[bot]	cf345d5f38	Bump urllib3 from 1.26.9 to 1.26.17 in /examples/research_projects/decision_transformer (#26554 ) Bump urllib3 in /examples/research_projects/decision_transformer Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.9 to 1.26.17. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.9...1.26.17) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-10-03 08:55:12 +02:00
dependabot[bot]	6de6fdd06d	Bump urllib3 from 1.26.5 to 1.26.17 in /examples/research_projects/visual_bert (#26552 ) Bump urllib3 in /examples/research_projects/visual_bert Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.5 to 1.26.17. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.5...1.26.17) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-10-03 08:55:01 +02:00
dependabot[bot]	e092b4ad68	Bump urllib3 from 1.26.5 to 1.26.17 in /examples/research_projects/lxmert (#26551 ) Bump urllib3 in /examples/research_projects/lxmert Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.5 to 1.26.17. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.5...1.26.17) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-10-03 08:54:50 +02:00
Sanchit Gandhi	68e85fc822	[Flax Examples] Seq2Seq ASR Fine-Tuning Script (#21764 ) * from seq2seq speech * [Flax] Example script for speech seq2seq * tests and fixes * make style * fix: label padding tokens * fix: label padding tokens over list * update ln names for Whisper * try datasets iter loader * create readme and append results * style * make style * adjust lr * use pt dataloader * make fast * pin gen max len * finish * add pt to requirements for test * fix pt -> torch * add accelerate	2023-09-29 16:42:58 +01:00
Phuc Van Phan	ba47efbfe4	docs: change assert to raise and some small docs (#26232 ) * docs: change assert to raise and some small docs * docs: add rule and some document * fix: fix bug * fix: fix bug * chorse: revert logging * chorse: revert	2023-09-28 10:14:17 +02:00
Phuc Van Phan	910faa3e1f	feat: adding num_proc to load_dataset (#26326 ) * feat: adding num_proc to load_dataset * feat: add add_num_proc for run_mlm_flax * feat: add num_proc for bart and t5 * chorse: remove	2023-09-22 19:22:47 +02:00
Nino Risteski	7d4e0c23c8	Update README.md (#26198 ) Fixed a few typos	2023-09-19 00:02:50 +02:00
Phuc Van Phan	8b5da9fc6e	refactor: change default block_size in block size > max position embeddings (#26069 ) * refactor: change default block_size when not initialize * reformat: add the min of block size	2023-09-18 16:47:57 +01:00
Phuc Van Phan	4fb64e285a	chore: correct update_step and correct gradient_accumulation_steps (#26068 )	2023-09-12 18:31:23 +01:00
Phuc Van Phan	5af2c62696	docs: add space to docs (#26067 ) * docs: add space to docs * docs: remove reduntant space	2023-09-11 22:03:26 +01:00
Phuc Van Phan	9cebae64ad	docs: update link huggingface map (#26077 )	2023-09-11 12:57:04 +01:00
Nino Risteski	f6295c6c53	Update README.md (#26003 ) fixed a typo	2023-09-06 10:55:11 +01:00
Joao Gante	9a70d6e56f	Trainer: delegate default generation values to `generation_config` (#25987 )	2023-09-05 14:47:00 +01:00
Traun Leyden	7011cd8667	Update RAG README.md with correct path to examples/seq2seq (#25953 ) Update README.md with correct path to examples/seq2seq	2023-09-05 12:31:59 +01:00
Susnato Dhar	404ff8fc17	Fix typo (#25966 ) * Update feature_extraction_clap.py * changed all lenght to length	2023-09-05 10:12:25 +02:00
Lysandre	d8e13b3e04	v4.34.dev.0	2023-09-04 15:12:11 -04:00
Zach Mueller	be0e189bd3	Revert frozen training arguments (#25903 ) * Revert frozen training arguments * TODO	2023-09-01 11:24:12 -04:00
Younes Belkada	4b79697865	🚨🚨🚨 [`Refactor`] Move third-party related utility files into `integrations/` folder 🚨🚨🚨 (#25599 ) * move deepspeed to `lib_integrations.deepspeed` * more refactor * oops * fix slow tests * Fix docs * fix docs * addess feedback * address feedback * final modifs for PEFT * fixup * ok now * trigger CI * trigger CI again * Update docs/source/en/main_classes/deepspeed.md Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * import from `integrations` * address feedback * revert removal of `deepspeed` module * revert removal of `deepspeed` module * fix conflicts * ooops * oops * add deprecation warning * place it on the top * put `FutureWarning` * fix conflicts with not_doctested.txt * add back `bitsandbytes` module with a depr warning * fix * fix * fixup * oops * fix doctests --------- Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2023-08-25 17:13:34 +02:00
Phuc Van Phan	656e17f6f7	correct resume training steps number in progress bar (#25691 ) feat: correct update resume update with steps	2023-08-23 20:09:14 +02:00
Matt	62396cff46	TF 2.14 compatibility (#25630 ) * Update the TF pin and see if anything breaks * make fixup * make fixup * make fixup	2023-08-22 13:13:38 +01:00
Sylvain Gugger	5c67682b16	v4.33.0.dev0	2023-08-21 07:07:04 -04:00
Zach Mueller	ca51499248	Make training args fully immutable (#25435 ) * Make training args fully immutable * Working tests, PyTorch * In test_trainer * during testing * Use proper dataclass way * Fix test * Another one * Fix tf * Lingering slow * Exception * Clean	2023-08-15 11:47:47 -04:00
dependabot[bot]	7a94ea4c64	Bump tornado from 6.3.2 to 6.3.3 in /examples/research_projects/lxmert (#25511 ) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3) --- updated-dependencies: - dependency-name: tornado dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-08-15 08:52:30 +02:00
dependabot[bot]	2552b8c5bd	Bump tornado from 6.3.2 to 6.3.3 in /examples/research_projects/visual_bert (#25512 ) Bump tornado in /examples/research_projects/visual_bert Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3) --- updated-dependencies: - dependency-name: tornado dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-08-15 08:52:20 +02:00
dependabot[bot]	0ebe7ae160	Bump gitpython from 3.1.30 to 3.1.32 in /examples/research_projects/decision_transformer (#25467 ) Bump gitpython in /examples/research_projects/decision_transformer Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.32. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.32) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-08-13 19:47:16 +02:00
dependabot[bot]	2b22cde71e	Bump gitpython from 3.1.30 to 3.1.32 in /examples/research_projects/distillation (#25468 ) Bump gitpython in /examples/research_projects/distillation Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.32. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.32) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-08-13 19:47:04 +02:00
Gema Parreño	0acf56224b	Update run_translation.py broken link example Pytoch (#25461 ) * Update run_translation.py Fixed link * Update run_translation.py	2023-08-11 15:41:24 +02:00
Yih-Dar	9c7b744795	Fix missing usage of `token` (#25382 ) * add missing tokens * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-08-08 16:27:24 +02:00
Zach Mueller	01ab39b65f	Load state in else (#25318 ) * Load else * New approach * Propagate	2023-08-08 05:41:00 -04:00
Phuc Van Phan	5fe36970e5	Adding more information in help parser on train_file and validation_file (#25324 ) chorse: adding new doc on train and val	2023-08-07 17:56:13 +02:00
Jackmin801	145109382a	Allow `trust_remote_code` in example scripts (#25248 ) * pytorch examples * pytorch mim no trainer * cookiecutter * flax examples * missed line in pytorch run_glue * tensorflow examples * tensorflow run_clip * tensorflow run_mlm * tensorflow run_ner * tensorflow run_clm * pytorch example from_configs * pytorch no trainer examples * Revert "tensorflow run_clip" This reverts commit `261f86ac1f`. * fix: duplicated argument	2023-08-07 16:32:25 +02:00
Yih-Dar	149cb0cce2	Add `token` arugment in example scripts (#25172 ) * fix * fix * fix * fix * fix * fix * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-08-02 11:17:31 +02:00
Yih-Dar	d53b8ad780	Update `use_auth_token` -> `token` in example scripts (#25167 ) * pytorch examples * tensorflow examples * flax examples --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-07-28 15:33:45 +02:00
Alan Ji	afa96fffdf	make run_generation more generic for other devices (#25133 ) * make run_generation more generic for other devices * use Accelerate to support any device type it supports. * make style * fix error usage of accelerator.prepare_model * use `PartialState` to make sure everything is running on the right device --------- Co-authored-by: statelesshz <jihuazhong1@huawei.com>	2023-07-28 08:20:10 -04:00
Lucain	6232c380f2	Fix `.push_to_hub` and cleanup `get_full_repo_name` usage (#25120 ) * Fix .push_to_hub and cleanup get_full_repo_name usage * Do not rely on Python bool conversion magic * request changes	2023-07-28 11:40:08 +02:00
dependabot[bot]	f1deb21fce	Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/visual_bert (#25097 ) Bump certifi in /examples/research_projects/visual_bert Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-25 17:25:14 -04:00
dependabot[bot]	45bde362d2	Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/decision_transformer (#25098 ) Bump certifi in /examples/research_projects/decision_transformer Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-25 17:25:05 -04:00
dependabot[bot]	6b8dbc283c	Bump certifi from 2022.12.7 to 2023.7.22 in /examples/research_projects/lxmert (#25096 ) Bump certifi in /examples/research_projects/lxmert Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](https://github.com/certifi/python-certifi/compare/2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-25 17:24:50 -04:00
Alan Ji	c879318cc5	replace `per_gpu_eval_batch_size` with `per_device_eval_batch_size` in readme of multiple-choice task (#25078 ) replace `per_gpu_eval_batch_size` with `per_device_eval_batch_size` in readme of multiple-choice	2023-07-25 08:11:56 -04:00
Zach Mueller	aa1b09c5d1	Change logic for logging in the examples (#24956 ) Change logic	2023-07-20 12:30:10 -04:00
dependabot[bot]	9f912ef62a	Bump aiohttp from 3.8.1 to 3.8.5 in /examples/research_projects/decision_transformer (#24954 ) Bump aiohttp in /examples/research_projects/decision_transformer Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.8.1 to 3.8.5. - [Release notes](https://github.com/aio-libs/aiohttp/releases) - [Changelog](https://github.com/aio-libs/aiohttp/blob/v3.8.5/CHANGES.rst) - [Commits](https://github.com/aio-libs/aiohttp/compare/v3.8.1...v3.8.5) --- updated-dependencies: - dependency-name: aiohttp dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-20 12:17:38 -04:00
dependabot[bot]	9859806608	Bump pygments from 2.11.2 to 2.15.0 in /examples/research_projects/decision_transformer (#24949 ) Bump pygments in /examples/research_projects/decision_transformer Bumps [pygments](https://github.com/pygments/pygments) from 2.11.2 to 2.15.0. - [Release notes](https://github.com/pygments/pygments/releases) - [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES) - [Commits](https://github.com/pygments/pygments/compare/2.11.2...2.15.0) --- updated-dependencies: - dependency-name: pygments dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-20 07:43:48 -04:00
statelesshz	37d8611ac9	replace no_cuda with use_cpu in test_pytorch_examples (#24944 ) * replace no_cuda with use_cpu in test_pytorch_examples * remove codes that never be used * fix style	2023-07-20 07:09:04 -04:00
ranchlai	8fd8c8e49e	Add multi-label text classification support to pytorch example (#24770 ) * Add text classification example * set the problem type and finetuning task * ruff reformated * fix bug for unseting label_to_id for regression * update README.md * fixed finetuning task * update comment * check if label exists in feature before removing * add useful logging	2023-07-20 07:02:44 -04:00
Hwijeen Ahn	dd49404a89	check if eval dataset is dict (#24877 ) * check if eval dataset is dict * formatting	2023-07-18 13:33:41 -04:00
Sylvain Gugger	e9ad51306f	4.32.0.dev0	2023-07-17 13:30:44 -04:00
dependabot[bot]	38dfb86958	Bump cryptography from 41.0.0 to 41.0.2 in /examples/research_projects/decision_transformer (#24833 ) Bump cryptography in /examples/research_projects/decision_transformer Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.0 to 41.0.2. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/41.0.0...41.0.2) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-07-17 07:17:17 -04:00
Bauke Brenninkmeijer	fc9e387dc0	Replacement of 20 asserts with exceptions (#24757 ) * initial replacements of asserts with errors/exceptions * replace assert with exception in generation, align and bart * reset formatting change * reset another formatting issue * Apply suggestion Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * don't touch this file * change to 'is not False' * fix type --------- Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2023-07-12 07:45:09 -04:00
Gema Parreño	4b26a61631	Fix loading dataset docs link in run_translation.py example (#24594 ) * fix loading dataset link * Update examples/tensorflow/translation/run_translation.py Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> * Update examples/tensorflow/translation/run_translation.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> --------- Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>	2023-07-03 15:21:21 +01:00
Jeroen Van Goey	fd8dcd0953	Udate link to RunHouse hardware setup documentation. (#24590 ) * Udate link to RunHouse hardware setup documentation. * Fix link to hardware setup in other location as well	2023-06-30 12:11:58 +01:00
Xiaoli Wang	239ace152b	Fix TypeError: Object of type int64 is not JSON serializable (#24340 ) * Fix TypeError: Object of type int64 is not JSON serializable * Convert numpy.float64 and numpy.int64 to float and int for json serialization * Black reformatted examples/pytorch/token-classification/run_ner_no_trainer.py * * make style	2023-06-27 12:15:49 +01:00
Matt	8e164c5400	Improved keras imports (#24448 ) * An end to accursed version-specific imports * No more K.is_keras_tensor() either * Update dependency tables * Use a cleaner call context function getter * Add a cap to <2.14 * Add cap to examples requirements too	2023-06-23 19:09:34 +01:00
Patrick von Platen	1609a436ec	Add MMS CTC Fine-Tuning (#24281 ) * Add mms ctc fine tuning * make style * More fixes that are needed * make fix-copies * make draft for README * add new file * move to new file * make style * make style * add quick test * make style * make style	2023-06-15 01:10:27 +02:00
Ethan	f7d80cb3d2	Fix steps bugs in no trainer examples (#24197 ) Fix step bugs in no trainer + load checkpoint + grad acc	2023-06-12 11:49:55 -04:00
Sylvain Gugger	ba695c1efd	v4.31.0.dev0	2023-06-07 16:49:00 -04:00
Zachary Mueller	cbf6bc2350	Oops, missed one (#24054 ) Oops	2023-06-06 13:30:19 -04:00
Zachary Mueller	072188d638	Act on deprecations in Accelerate no_trainer examples (#24053 ) Act on deprecation	2023-06-06 13:04:38 -04:00
dependabot[bot]	539e2281cd	Bump cryptography from 39.0.1 to 41.0.0 in /examples/research_projects/decision_transformer (#23964 ) Bump cryptography in /examples/research_projects/decision_transformer Bumps [cryptography](https://github.com/pyca/cryptography) from 39.0.1 to 41.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/39.0.1...41.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-06-02 16:23:44 -04:00
Matt	167a0d8f87	Add an option to reduce compile() console spam (#23938 ) * Add an option to reduce compile() console spam * Add annotations to the example scripts * Add notes to the quicktour docs as well * minor fix	2023-06-02 15:28:52 +01:00
Sylvain Gugger	3ff443a6d9	Re-enable squad test (#23912 ) * Re-enable squad test * [all-test] * [all-test] Fix all test command * Fix the all-test	2023-05-31 13:44:26 -04:00
Sylvain Gugger	00f6ba0e7e	Skip failing test for now	2023-05-31 06:31:33 -04:00
Ran Ran	e724246935	Fix no such file or directory error (#23783 ) * Fix no such file or directory error * Address comment * Fix formatting issue	2023-05-26 14:24:57 -04:00
dependabot[bot]	d685e330b5	Bump tornado from 6.0.4 to 6.3.2 in /examples/research_projects/visual_bert (#23767 ) Bump tornado in /examples/research_projects/visual_bert Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.0.4 to 6.3.2. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.0.4...v6.3.2) --- updated-dependencies: - dependency-name: tornado dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-25 16:16:12 -04:00
dependabot[bot]	4b0e7ded1c	Bump tornado from 6.0.4 to 6.3.2 in /examples/research_projects/lxmert (#23766 ) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.0.4 to 6.3.2. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.0.4...v6.3.2) --- updated-dependencies: - dependency-name: tornado dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-25 16:16:01 -04:00
Sylvain Gugger	6e4bc67099	Revamp test selection for the example tests (#23737 ) * Revamp test selection for the example tests * Rename old XLA test and fake modif in run_glue * Fixes * Fake Trainer modif * Remove fake modifs	2023-05-25 09:38:21 -04:00
Wang, Yi	33687a3f61	add GPTJ/bloom/llama/opt into model list and enhance the jit support (#23291 ) Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>	2023-05-24 10:57:56 +01:00
zspo	003a0cf8cc	Fix some docs what layerdrop does (#23691 ) * Fix some docs what layerdrop does * Update src/transformers/models/data2vec/configuration_data2vec_audio.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Fix more docs --------- Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2023-05-23 14:50:40 -04:00
dependabot[bot]	aa30cd4f3f	Bump requests from 2.22.0 to 2.31.0 in /examples/research_projects/lxmert (#23668 ) Bump requests in /examples/research_projects/lxmert Bumps [requests](https://github.com/psf/requests) from 2.22.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](https://github.com/psf/requests/compare/v2.22.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-23 05:31:53 -04:00
dependabot[bot]	9bf72ae564	Bump requests from 2.22.0 to 2.31.0 in /examples/research_projects/visual_bert (#23670 ) Bump requests in /examples/research_projects/visual_bert Bumps [requests](https://github.com/psf/requests) from 2.22.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](https://github.com/psf/requests/compare/v2.22.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-23 05:31:30 -04:00
dependabot[bot]	ecc05f8c1e	Bump requests from 2.27.1 to 2.31.0 in /examples/research_projects/decision_transformer (#23673 ) Bump requests in /examples/research_projects/decision_transformer Bumps [requests](https://github.com/psf/requests) from 2.27.1 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](https://github.com/psf/requests/compare/v2.27.1...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-23 05:28:09 -04:00
Zachary Mueller	b191d7db44	Update all no_trainer with skip_first_batches (#23664 )	2023-05-22 14:49:31 -04:00
sshahrokhi	6f72e71f97	changing the requirements to a cpu torch version that works (#23483 )	2023-05-22 12:58:55 -04:00
Boda Sadallah	a7920065f2	fix bug in group_texts function, that was inserting short batches (#23429 ) * fix bug in group_texts function, that was inserting short batches * fully exclude short batches and return empty dict instead * fix style	2023-05-18 14:22:30 -04:00
Zachary Mueller	8a58809312	Fix translation no_trainer (#23407 ) * Fix translation	2023-05-16 13:10:42 -04:00
Yih-Dar	d51296d9c2	skip `test_run_squad_no_trainer` for now (#23302 ) skip Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2023-05-11 19:26:48 +02:00
Hari	5d02e6bd20	Convert numpy arrays to lists before saving the evaluation metrics as json (#23268 ) * convert numpy array to list before writing to json per_category_iou and per_category_accuracy are ndarray in the eval_metrics * code reformatted with make style	2023-05-11 08:54:23 -04:00
Maria Khalusova	91f4c84a19	CTC example: updated trainer parameters to save tokenizer (#23243 ) trainer parameters changed to save tokenizer in addition to feature_extractor	2023-05-10 07:45:10 -04:00
Sylvain Gugger	a0c0a78233	v4.30.0.dev0	2023-05-09 14:59:38 -04:00
Nicolas Patry	c34a525d2f	Proposed fix for TF example now running on safetensors. (#23208 ) * Proposed fix for TF example now running on safetensors. * Adding more warnings and returning keys. * Trigger CI * Trigger CI --------- Co-authored-by: Sylvain Gugger <Sylvain.gugger@gmail.com>	2023-05-09 13:04:27 -04:00
Sebastian	1a8f61110e	fix: Update run_qa.py to work with deepset/germanquad (#23225 ) Call str on id to make sure any ints are converted into the expected format for squad datasets	2023-05-09 09:20:10 -04:00
Sylvain Gugger	fd6970bc56	Skip failing test	2023-05-08 08:52:44 -04:00
Ashwin Mathur	fc6c8b0eaa	Add `no_trainer` scripts to pre-train Vision Transformers (#23156 ) * Add run_mim_no_trainer.py draft from #20412 Add parse_args method and copy over other dependencies Add Method call for sending telemetry Initialize Accelerator Make one log on every process Set seed and Handle repository creation Initialize dataset and Set validation split Create Config Adapt Config Update Config Create Feature Extractor Create model Set column names Create transforms Create mask generator Create method to preprocess images Shuffle datasets if needed and set transforms Create Dataloaders Add optimizer Add learning rate scheduler Prepare everything with our accelerator Tie weights for TPU training Recalculate training steps and training epochs Set accelerator checkpointing steps Initialize trackers and store configuration Set total batch size Fix typo: mlm -> mim Log info at the start of training Load in the weights and states from previous save update the progress_bar if load from checkpoint Define train loop Add evaluation loop to training Add to parse_args method Push repo to hub Save accelerator state End training and save model and feature extractor Remove unused imports Fix trailing whitespace * Update code based on comments, Rename feature_extractor to image_processor * Fix linting * Add argument for learning rate * Add argument for setting number of training epochs * Remove incorrect logger argument * Convert max_train_steps to int for tqdm --------- Co-authored-by: Saad Mahmud <shuvro.mahmud79@gmail.com>	2023-05-05 13:22:49 -04:00
Robert Stone	b6933d76d2	Tidy Pytorch GLUE benchmark example (#23134 ) Migration to Evaluate for metric is not quite complete	2023-05-03 15:50:41 -04:00
Alex Punnen	805db1fe13	num_noise_spans should be <= num_items #22246 (#22938 )	2023-05-02 13:07:30 -04:00
regisss	bcedd0a471	Save the tokenizer and image preprocessor after training a model with the contrastive image-text example (#23035 ) Save tokenizer and image preprocessor	2023-05-02 09:23:16 -04:00
dependabot[bot]	b8648290d2	Bump flask from 2.0.3 to 2.3.2 in /examples/research_projects/decision_transformer (#23094 ) Bump flask in /examples/research_projects/decision_transformer Bumps [flask](https://github.com/pallets/flask) from 2.0.3 to 2.3.2. - [Release notes](https://github.com/pallets/flask/releases) - [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/flask/compare/2.0.3...2.3.2) --- updated-dependencies: - dependency-name: flask dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-05-01 20:15:11 -04:00
Lingepumpe	5427250351	Avoid invalid escape sequences, use raw strings (#22936 ) * Avoid invalid escape sequences, use raw strings * Integrate PR feedback	2023-04-25 09:17:56 -04:00
Roy Hvaara	874c7caf19	Remove broken test_data symlink in legacy s2s examples (#22876 )	2023-04-21 15:35:42 +01:00
Sayak Paul	4116d1ec75	[Examples/TensorFlow] minor refactoring to allow compatible datasets to work (#22879 ) minor refactoring to allow compatible datasets to work.	2023-04-20 18:21:01 +05:30
Zachary Mueller	cd3e0211a6	Remove accelerate from tf test reqs (#22777 ) Remove accelerate from tf	2023-04-17 12:31:21 -04:00
Matt	2237127a6c	Fix sneaky torch dependency in TF example (#22804 )	2023-04-17 16:11:52 +01:00
Sayak Paul	390e121fb5	[Examples] TPU-based training of a language model using TensorFlow (#21657 ) * add: tokenizer training script for TF TPU LM training. * add: script for preparing the TFRecord shards. * add: sequence of execution to readme. * remove limit from the tfrecord shard name. * Add initial train_model.py * Add basic training arguments and model init * Get up to the point of writing the data collator * Pushing progress so far! * Complete first draft of model training code * feat: grouping of texts efficiently. Co-authored-by: Matt <rocketknight1@gmail.com> * Add proper masking collator and get training loop working * fix: things. * Read sample counts from filenames * Read sample counts from filenames * Draft README * Improve TPU warning * Use distribute instead of distribute.experimental * Apply suggestions from code review Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> * Modularize loading and add MLM probability as arg * minor refactoring to better use the cli args. * readme fillup. * include tpu and inference sections in the readme. * table of contents. * parallelize maps. * polish readme. * change script name to run_mlm.py * address PR feedback (round I). --------- Co-authored-by: Matt <rocketknight1@gmail.com> Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>	2023-04-14 10:41:01 +05:30
Sylvain Gugger	888c4a2ae0	v4.29.0.dev0	2023-04-12 20:04:29 -04:00
Sylvain Gugger	1b1867d86b	Replace -100s in predictions by the pad token (#22693 ) * Replace -100s in predictions by the pad token * Style * Try to catch them all	2023-04-11 09:32:20 -04:00
Mikel Penagarikano	d5239bab5b	Sync preprocesses before loading the processor at run_speech_recognition_ctc.py (#21926 ) * Update run_speech_recognition_ctc.py Make sure all processes wait until data is saved before loading the processor from the output_dit * Make sure all processes wait until data is saved before loading the processor from the output_dit * Update run_speech_recognition_ctc.py * Update run_speech_recognition_seq2seq.py	2023-04-05 09:36:04 -04:00
Maziyar Panahi	98268b2e76	Add id2label and label2id to model's config in run_xnil (#22558 ) Add id2label and label2id to config in run_xnil	2023-04-04 09:28:57 -04:00
dependabot[bot]	6fc44656b4	Bump redis from 4.5.3 to 4.5.4 in /examples/research_projects/decision_transformer (#22494 ) Bump redis in /examples/research_projects/decision_transformer Bumps [redis](https://github.com/redis/redis-py) from 4.5.3 to 4.5.4. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](https://github.com/redis/redis-py/compare/v4.5.3...v4.5.4) --- updated-dependencies: - dependency-name: redis dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-03-31 10:50:33 -04:00
Sabine	173193ccd0	Update Neptune docs (#22452 )	2023-03-29 13:15:38 -04:00
dependabot[bot]	32ff06403d	Bump redis from 4.1.4 to 4.5.3 in /examples/research_projects/decision_transformer (#22410 ) Bump redis in /examples/research_projects/decision_transformer Bumps [redis](https://github.com/redis/redis-py) from 4.1.4 to 4.5.3. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](https://github.com/redis/redis-py/compare/v4.1.4...v4.5.3) --- updated-dependencies: - dependency-name: redis dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-03-27 20:23:55 -04:00
Sylvain Gugger	057e1d7473	Fix quality	2023-03-27 13:17:14 -04:00
Donny Greenberg	f02e3a2b18	Hardware Auto-Setup for Examples (#22319 ) * Add initial remote hardware auto-setup docs * Fix a few typos and clarify some language * Add missing dependency * Update self-hosted launch script with Sylvain's comments. * Formatting. * Trigger CI * Style	2023-03-27 13:07:53 -04:00
Joao Gante	88dae78f4d	TensorFlow: pin maximum version to 2.12 (#22364 )	2023-03-24 18:45:03 +00:00
Sylvain Gugger	6587125c0a	Pin tensorflow-text to go with tensorflow (#22362 ) * Pin tensorflow-text to go with tensorflow * Make it more convenient to pin TensorFlow * setup don't like f-strings	2023-03-24 10:54:06 -04:00
Sylvain	ef28df0572	Fix quality due to ruff release	2023-03-22 20:45:08 -04:00
Connor Henderson	8e6c34b390	fix: Allow only test_file in pytorch and flax summarization (#22293 ) allow only test_file in pytorch and flax summarization	2023-03-22 10:46:56 +00:00
Wang, Yi	4ccaf268fb	add low_cpu_mem_usage option in run_clm.py example which will benefit… (#22288 ) * add low_cpu_mem_usage option in run_clm.py example which will benefit LLM loading Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * update all the example and README under language-modeling Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> --------- Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>	2023-03-22 10:42:39 +00:00
jiqing-feng	8472a224fb	Enable traced model for text-generation task (#22265 )	2023-03-22 10:19:26 +00:00
Sylvain Gugger	ebdb185bef	v4.28.0.dev0	2023-03-14 13:49:10 -04:00
bofeng huang	6192549c1f	[examples/speech-recognition] Add SpecAugment to run_speech_recognition_seq2seq.py (#21942 ) * Add specaugment to run_speech_recognition_seq2seq.py * Remove useless argument: text_column * Fix quality * Update return_attention_mask condition * Update specaugment arguments only for whisper models * Remove SpecAugment arguments from ModelArguments, only leave default values for simplicity * Apply suggestions from code review Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * Update apply_spec_augment only for whisper models * Apply suggestions from code review Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * Rename return_attention_mask to forward_attention_mask to avoid confusion with wav2vec2 models --------- Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>	2023-03-08 17:59:31 +01:00
Matt	d128f2ffab	Stop requiring Torch for our TF examples! (#21997 ) * Stop requiring Torch for our TF examples! * Slight tweak to logging in the example itself	2023-03-07 15:54:10 +00:00
Matt	5d8efc79db	Add TF contrastive image text finetuning example (#21939 ) * Initial commit * stash commit * Add model checkpointing and pushing * Fix model name inference * Update README * Update README * Remove a couple of Torch references * Update copyright date * make fixup * Update PushToHubCallback args! * Remove the torch summary * Add strategy.scope	2023-03-06 16:57:40 +00:00
Matt	1d3a1cc44b	Add check for different embedding types in examples (#21881 ) * Add check for different embedding types in examples * Correctly update summarization example	2023-03-01 16:57:06 +00:00
bofeng huang	3c0ce60855	[examples/summarization] deal with `max_length` and `num_beams` (#21740 ) * Override the decoding parameters of Seq2SeqTrainer * Fix quality * Fix max_length parameter * Fix quality * Remove redundant parameter max_length * Separate the preprocess of train and validation to use different max_target_length	2023-02-27 08:18:14 +01:00
Sanchit Gandhi	13489248fa	[Examples] Generalise run audio classification for log-mel models (#21756 ) * [Examples] Generalise run audio classification for log-mel models * batch feature extractor * make style	2023-02-24 09:19:07 +01:00
Sylvain Gugger	b19d64d852	Respect documentation on passive log level (#21700 ) * Respect documentation on passive log level * Fix test and set log level in examples * Add doc	2023-02-22 09:39:18 +01:00
Aaron Gokaslan	5e8c8eb5ba	Apply ruff flake8-comprehensions (#21694 )	2023-02-22 09:14:54 +01:00
Arthur	4194e5f42b	Fix-rag-finetune-project-requirement (#21697 ) pin pytorch lightning requirement	2023-02-20 17:23:39 +01:00
dependabot[bot]	fcfd4ec789	Bump werkzeug from 2.0.3 to 2.2.3 in /examples/research_projects/decision_transformer (#21658 ) Bump werkzeug in /examples/research_projects/decision_transformer Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.0.3 to 2.2.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/2.0.3...2.2.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-02-16 09:23:43 -05:00
regisss	751f17aa48	Fix typos in contrastive-image-text example README (#21665 )	2023-02-16 09:10:25 -05:00
Warren Green	fd5320bb57	Add missing arguemtn to run_clip.py (#21588 )	2023-02-13 10:27:23 -05:00
dependabot[bot]	92487f5d0b	Bump ipython from 8.1.1 to 8.10.0 in /examples/research_projects/decision_transformer (#21577 ) Bump ipython in /examples/research_projects/decision_transformer Bumps [ipython](https://github.com/ipython/ipython) from 8.1.1 to 8.10.0. - [Release notes](https://github.com/ipython/ipython/releases) - [Commits](https://github.com/ipython/ipython/compare/8.1.1...8.10.0) --- updated-dependencies: - dependency-name: ipython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-02-13 10:21:50 -05:00
steventk-g	c88b11c591	Add _mp_fn to run_mae.py for XLA testing (#21551 ) Update run_mae.py	2023-02-10 09:53:55 -05:00
lee1jun	b31cee6727	fix typo in run_speech_recognition_ctc.py (#21528 ) Update run_speech_recognition_ctc.py There should be `# limitations under the License` line at the end of the documentation section.	2023-02-09 09:46:40 -05:00
Stefan Schweter	d3046dad80	[Doc] Minor URL fixes in PyTorch Text Classification Readme (#21511 ) docs: fix some references in PyTorch text classification readme	2023-02-08 09:39:52 -05:00
dependabot[bot]	e024cd715e	Bump cryptography from 36.0.2 to 39.0.1 in /examples/research_projects/decision_transformer (#21507 ) Bump cryptography in /examples/research_projects/decision_transformer Bumps [cryptography](https://github.com/pyca/cryptography) from 36.0.2 to 39.0.1. - [Release notes](https://github.com/pyca/cryptography/releases) - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/36.0.2...39.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-02-08 09:25:06 -05:00
Sylvain Gugger	67d074874d	Cleanup quality (#21493 ) * Remove mentions of flake8/isort * Clean up inits * Deall with all other inits * Last special rule for dummy files	2023-02-07 12:27:31 -05:00
Jeroen Van Der Donckt	bbe98ea9c3	🖊️ fix typo in pytorch semantic segmentation readme (#21492 )	2023-02-07 09:39:24 -05:00
dependabot[bot]	35f93f299f	Bump oauthlib from 3.2.1 to 3.2.2 in /examples/research_projects/decision_transformer (#21481 ) Bump oauthlib in /examples/research_projects/decision_transformer Bumps [oauthlib](https://github.com/oauthlib/oauthlib) from 3.2.1 to 3.2.2. - [Release notes](https://github.com/oauthlib/oauthlib/releases) - [Changelog](https://github.com/oauthlib/oauthlib/blob/master/CHANGELOG.rst) - [Commits](https://github.com/oauthlib/oauthlib/compare/v3.2.1...v3.2.2) --- updated-dependencies: - dependency-name: oauthlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-02-06 18:27:14 -05:00
Sylvain Gugger	6f79d26442	Update quality tooling for formatting (#21480 ) * Result of black 23.1 * Update target to Python 3.7 * Switch flake8 to ruff * Configure isort * Configure isort * Apply isort with line limit * Put the right black version * adapt black in check copies * Fix copies	2023-02-06 18:10:56 -05:00
Stas Bekman	3b9a1dc132	[examples] improve block_size warning message (#21463 )	2023-02-06 08:36:12 -08:00
Kaustubh Dhole	182afb7dc6	Fixed RAG script which was failing on dummy example (#21416 ) * do not use prefix="val" for test The dummy example fails when test_epoch_end is called. The prefix="test" should be dynamic in the log metrics too. * Create test.source * Create test.target	2023-02-06 09:27:34 -05:00
Erwann Millon	ea55bd86b9	Add VQGAN-CLIP research project (#21329 ) * Add VQGAN-CLIP research project * fixed style issues * Update examples/research_projects/vqgan-clip/README.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/VQGAN_CLIP.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/requirements.txt Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/README.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/VQGAN_CLIP.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/VQGAN_CLIP.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/VQGAN_CLIP.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update examples/research_projects/vqgan-clip/loaders.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * replace CLIPProcessor with tokenizer, change asserts to exceptions * rm unused import * remove large files (jupyter notebook linked in readme, imgs migrated to hf dataset) * add tokenizers dependency * Remove comment Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * rm model checkpoints --------- Co-authored-by: Erwann Millon <erwann@Erwanns-MacBook-Air.local> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>	2023-02-02 14:45:35 -05:00
amyeroberts	e5db7051a8	Add TF image classification example script (#19956 ) * TF image classification script * Update requirements * Fix up * Add tests * Update test fetcher Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Fix directory path * Adding `zero-shot-object-detection` pipeline doctest. (#20274) * Adding `zero-shot-object-detection` pipeline doctest. * Remove nested_simplify. * Add generate kwargs to `AutomaticSpeechRecognitionPipeline` (#20952) * Add generate kwargs to AutomaticSpeechRecognitionPipeline * Add test for generation kwargs * Trigger CI * Data collator returns np * Update feature extractor -> image processor * Bug fixes - updates to reflect changes in API * Update flags to match PT & run faster * Update instructions - Maria's comment * Update examples/tensorflow/image-classification/README.md * Remove slow decorator --------- Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com> Co-authored-by: bofeng huang <bofenghuang7@gmail.com> Co-authored-by: Sylvain Gugger <Sylvain.gugger@gmail.com>	2023-02-01 19:09:36 +00:00
Quentin Lhoest	074d6b75fd	Simplify column_names in run_clm/mlm (#21382 ) * simplify column_names in run_clm * simplify column_names in run_mlm * minor	2023-01-31 15:23:47 +01:00
Stas Bekman	98d88b23f5	[`run_(clm\|mlm).py` examples] add streaming dataset support (#21343 ) * [run_clm example] add streaming dataset support * unrefactor kwargs * fix * fix * require datasets>=2.0.0 * port to mlm	2023-01-30 14:01:35 -08:00
dependabot[bot]	36b668fa06	Bump onnx from 1.11.0 to 1.13.0 in /examples/research_projects/decision_transformer (#21331 ) Bump onnx in /examples/research_projects/decision_transformer Bumps [onnx](https://github.com/onnx/onnx) from 1.11.0 to 1.13.0. - [Release notes](https://github.com/onnx/onnx/releases) - [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md) - [Commits](https://github.com/onnx/onnx/compare/v1.11.0...v1.13.0) --- updated-dependencies: - dependency-name: onnx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-27 10:13:13 -05:00
Matt	071529bd54	Use return_tensors="np" instead of "tf" (#21266 ) Return NP instead of TF tensors for our data loading pipeline	2023-01-24 13:37:49 +00:00
Sylvain Gugger	7119bb052a	v4.27.0.dev0	2023-01-23 16:52:35 -05:00
Mostafa Elhoushi	5603f78fc4	Add scikit-learn dependency to train langage-modeling (#21229 )	2023-01-23 09:54:45 -05:00
amyeroberts	4bc18e7a83	Update examples with image processors (#21155 ) * Update examples to use image processors * Small fixes * Resolve conflicts	2023-01-19 15:14:58 +00:00
dependabot[bot]	f70ee51029	Bump future from 0.18.2 to 0.18.3 in /examples/research_projects/visual_bert (#21173 ) Bump future in /examples/research_projects/visual_bert Bumps [future](https://github.com/PythonCharmers/python-future) from 0.18.2 to 0.18.3. - [Release notes](https://github.com/PythonCharmers/python-future/releases) - [Changelog](https://github.com/PythonCharmers/python-future/blob/master/docs/changelog.rst) - [Commits](https://github.com/PythonCharmers/python-future/compare/v0.18.2...v0.18.3) --- updated-dependencies: - dependency-name: future dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-18 11:17:35 -05:00
dependabot[bot]	0194665c33	Bump future from 0.18.2 to 0.18.3 in /examples/research_projects/lxmert (#21169 ) Bumps [future](https://github.com/PythonCharmers/python-future) from 0.18.2 to 0.18.3. - [Release notes](https://github.com/PythonCharmers/python-future/releases) - [Changelog](https://github.com/PythonCharmers/python-future/blob/master/docs/changelog.rst) - [Commits](https://github.com/PythonCharmers/python-future/compare/v0.18.2...v0.18.3) --- updated-dependencies: - dependency-name: future dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-18 11:16:43 -05:00
Sylvain Gugger	05e72aa0c4	Adapt repository creation to latest hf_hub (#21158 ) * Adapt repository creation to latest hf_hub * Update all examples * Fix other tests, add Flax examples * Address review comments	2023-01-18 11:14:00 -05:00
Observer46	ff8dcb5efa	Fix arguments passed to predict function in QA Seq2seq training script (#21026 ) fix args passed to predict function	2023-01-06 07:19:42 -05:00
Roy Hvaara	35a7052b61	[NumPy] Remove references to deprecated NumPy type aliases (#21022 ) [NumPy] Remove references to deprecated NumPy type aliases. This change replaces references to a number of deprecated NumPy type aliases (np.bool, np.int, np.float, np.complex, np.object, np.str) with their recommended replacement (bool, int, float, complex, object, str). NumPy 1.24 drops the deprecated aliases, so we must remove uses before updating NumPy. Co-authored-by: Peter Hawkins <phawkins@google.com> Co-authored-by: Peter Hawkins <phawkins@google.com>	2023-01-05 13:02:10 -05:00
Magnus Pierrau	1d21471c78	Added mask_time_prob and mask_time_length arguments to wav2vec2 pretraining script (#20985 ) Added mask_time_prob and mask_time_length arguments to wav2vec2 pretraining script and readme - new branch	2023-01-05 16:24:55 +00:00
dependabot[bot]	b7417bee87	Bump gitpython from 3.0.2 to 3.1.30 in /examples/research_projects/distillation (#21011 ) Bump gitpython in /examples/research_projects/distillation Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.0.2 to 3.1.30. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.0.2...3.1.30) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-04 15:36:42 -05:00
dependabot[bot]	05b736c16e	Bump gitpython from 3.1.18 to 3.1.30 in /examples/research_projects/decision_transformer (#21010 ) Bump gitpython in /examples/research_projects/decision_transformer Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.18 to 3.1.30. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.18...3.1.30) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2023-01-04 15:36:33 -05:00
milyiyo	3b309818e7	Refactor the function get_results (#20999 )	2023-01-04 12:05:36 -05:00
Wang, Yi	9c9fe89f84	[run_clm example] add torch_dtype option for model load. (#20971 ) * [run_clm example] add torch_dtype option for model load. for BLOOM 175B model. peak memory will reduce about 350G for inference. the weight of BLOOM in model hub is bfloat16 Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> * add other type in option * fix style Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>	2023-01-03 09:33:11 -05:00
Alex Hedges	0b686a8a1e	Remove non-breaking spaces (#20929 ) * Remove non-breaking space in comment It was likely added unintionally. * Remove remaining non-breaking spaces	2022-12-29 02:12:40 -05:00
Márton Makrai	3830b3f74a	Fixes typo in the help text for --max_length (#20883 )	2022-12-24 02:07:06 -05:00
NielsRogge	d87e381f93	[Examples] Update big table (#20845 ) Update big table Co-authored-by: Niels Rogge <nielsrogge@Nielss-MBP.localdomain>	2022-12-21 11:34:31 +01:00
fzyzcjy	ae3cbbcaf6	Fix tiny typo (#20841 ) * Fix typo * Update README.md * Update run_mlm_flax_stream.py * Update README.md	2022-12-20 03:17:59 -05:00
dependabot[bot]	da95f6ca4c	Bump certifi in /examples/research_projects/visual_bert (#20673 ) Bumps [certifi](https://github.com/certifi/python-certifi) from 2020.6.20 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](https://github.com/certifi/python-certifi/compare/2020.06.20...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-12-08 11:15:42 -05:00
dependabot[bot]	efd7c021ee	Bump certifi in /examples/research_projects/decision_transformer (#20677 ) Bumps [certifi](https://github.com/certifi/python-certifi) from 2021.10.8 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](https://github.com/certifi/python-certifi/compare/2021.10.08...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-12-08 11:15:11 -05:00
dependabot[bot]	9e33e19bf5	Bump certifi in /examples/research_projects/lxmert (#20672 ) Bumps [certifi](https://github.com/certifi/python-certifi) from 2020.6.20 to 2022.12.7. - [Release notes](https://github.com/certifi/python-certifi/releases) - [Commits](https://github.com/certifi/python-certifi/compare/2020.06.20...2022.12.07) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-12-08 11:14:54 -05:00
Emmanuel Schmidbauer	0526a075c5	run_speech_recognition_seq2seq.py: add cache_dir param to dataset (#20540 )	2022-12-07 18:23:16 +00:00
Francisco Kurucz	f821bea0ad	Fix link to speech encoder decoder model in speech recognition readme (#20633 )	2022-12-06 15:46:41 -05:00
Wang, Yi	ae06bce888	exclude jit time from the speed metric calculation of evaluation and prediction (#20553 ) Signed-off-by: Wang, Yi A <yi.a.wang@intel.com> Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>	2022-12-06 07:37:01 -05:00
Sylvain Gugger	60d1f31bb0	v4.26.0.dev0	2022-12-01 16:19:33 -05:00
Wang, Yi	d752337baa	QnA example: add speed metric (#20522 )	2022-12-01 12:04:19 -05:00
Pi Esposito	fb2b45e562	add in layer gpt2 tokenizer (#20421 ) * add minimal working gpt2 tokenizer * graph mode and output equivalence tests working * not today tensorflow. serialization test passing! * fix style, documentation, docstrings and all that jazz * passing consistency checks * move keras nlp to tf dependencies * fix tf modeling utils and gpt2 attention to enable compiling * fix (I hope) keras nlp dependencies * rever changes on generation * remove debug prints * remove redundant tf dummy objects * add from config, get config and max length settings to address review * let flake ignore the error on distillation you are welcome * test from config * add padding test * address sgugger review	2022-11-29 10:02:40 -05:00
Katie Le	667ccea722	Replace assertion with ValueError exceptions in run_image_captioning_flax.py (#20365 ) * replace 4 asserts with ValueError exception for control flow * Update examples/flax/image-captioning/run_image_captioning_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * Update examples/flax/image-captioning/run_image_captioning_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * reformatted file * uninstalled trasformers and applied make style Co-authored-by: Bibi <Bibi@katies-mac.local> Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>	2022-11-28 15:06:25 +00:00
dependabot[bot]	624ae09f5c	Bump pillow in /examples/research_projects/decision_transformer (#20378 ) Bumps [pillow](https://github.com/python-pillow/Pillow) from 9.0.1 to 9.3.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/9.0.1...9.3.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-11-22 08:15:42 -05:00
Zachary Mueller	9d1ef009b8	Fix flakey test with seed (#20318 )	2022-11-18 11:33:25 -05:00
Sylvain Gugger	a3f7458066	Pin to the right version...	2022-11-18 07:12:55 -05:00
Sanchit Gandhi	c29a2f7c9c	[ASR Examples] Update README for Whisper (#20230 ) * [ASR Examples] Update README for seq2seq * add language info * add training results * re-word	2022-11-18 11:24:25 +00:00
Zachary Mueller	441811ecd7	Fix summarization script (#20286 )	2022-11-16 15:57:07 -05:00
Jiahao Li	9681f052a1	Fix result saving errors of pytorch examples (#20276 )	2022-11-16 09:51:04 -05:00
Zachary Mueller	822ae69c1b	Update reqs to include min gather_for_metrics Accelerate version (#20242 ) * Update reqs to include min gather_for_metrics Accelerate version * Other reqs	2022-11-15 13:28:00 -05:00
Muhammad Sakib Khan Inan	777b1bfe62	New logging support to "Trainer" Class (ClearML Logger) (#20184 ) * Init Update * ClearML Callbacks integration * update corrections * args reporting updated * {'tensorboard': False, 'pytorch': False} * ClearML Tests added * add clearml * output_uri=True in Task.init * reformatted integrations.py * reformatted and fixed * IF-ELSE statement issue on "has_clearml" resolved * Add clearml in main callback docs * Add additional clearml documentation * Update src/transformers/integrations.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Accept suggestion Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Accept suggestion Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Small change in comments * Make style clearml * Accept suggestion Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: Victor Sonck <victor.sonck@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2022-11-15 10:08:59 -05:00
Yih-Dar	cf7b98b807	Fix `run_clip.py` (#20234 ) Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2022-11-15 15:45:21 +01:00
Ming Liu	36b063ed4f	Update README.md (#20188 ) There is typo in the original hyperlink. Below is the original version: Based on the script [`run_translation_no_trainer.py`](https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/run_translationn_no_trainer.py).	2022-11-14 12:53:02 -05:00
Sanchit Gandhi	af1a7c8ca3	[Examples] Generalise Seq2Seq ASR to handle Whisper (#19519 ) * merge conflicts * bos and eos in datacollator * (temp) hardcode removal of attention mask * freeze encoder * actually freeze encoder * set max length / num beams according to gen kwargs * (temp) fix tests * don't pop attn mask * override return attention mask config from Hub * Hub configs updated 🤗 * final fixes * update type annotations * backward comp	2022-11-14 17:45:46 +00:00
Joao Gante	f270b960d6	Generate: move generation_.py src files into generation/.py (#20096 ) * move generation_.py src files into generation/.py * populate generation.__init__ with lazy loading * move imports and references from generation.xxx.object to generation.object	2022-11-09 15:34:08 +00:00
Tom Aarsen	b8112eddec	Replace unsupported facebookresearch/bitsandbytes (#20093 ) With https://github.com/TimDettmers/bitsandbytes, which is by the same author and is still being updated	2022-11-07 08:52:03 -05:00
bhuang	3502c202f9	Update README.md (#20063 )	2022-11-04 08:56:54 -04:00
Sylvain Gugger	06886d5a68	Only resize embeddings when necessary (#20043 ) * Only resize embeddings when necessary * Add comment	2022-11-03 12:05:04 -04:00
amyeroberts	a6b7759880	Add Image Processors (#19796 ) * Add CLIP image processor * Crop size as dict too * Update warning * Actually use logger this time * Normalize doesn't change dtype of input * Add perceiver image processor * Tidy up * Add DPT image processor * Add Vilt image processor * Tidy up * Add poolformer image processor * Tidy up * Add LayoutLM v2 and v3 imsge processors * Tidy up * Add Flava image processor * Tidy up * Add deit image processor * Tidy up * Add ConvNext image processor * Tidy up * Add levit image processor * Add segformer image processor * Add in post processing * Fix up * Add ImageGPT image processor * Fixup * Add mobilevit image processor * Tidy up * Add postprocessing * Fixup * Add VideoMAE image processor * Tidy up * Add ImageGPT image processor * Fixup * Add ViT image processor * Tidy up * Add beit image processor * Add mobilevit image processor * Tidy up * Add postprocessing * Fixup * Fix up * Fix flava and remove tree module * Fix image classification pipeline failing tests * Update feature extractor in trainer scripts * Update pad_if_smaller to accept tuple and int size * Update for image segmentation pipeline * Update src/transformers/models/perceiver/image_processing_perceiver.py Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com> * Update src/transformers/image_processing_utils.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update src/transformers/models/beit/image_processing_beit.py Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * PR comments - docstrings; remove accidentally added resize; var names * Update docstrings * Add exception if size is not in the right format * Fix exception check * Fix up * Use shortest_edge in tuple in script Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com> Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>	2022-11-02 11:57:36 +00:00
Sylvain Gugger	c3a93d8d82	v4.25.0.dev0	2022-10-31 21:48:40 -04:00
Sanchit Gandhi	f38a145418	[ASR] Update 'tasks' for model card (#19986 )	2022-10-31 16:50:17 +00:00
regisss	5d2d51a0fb	Fix LR (#19875 )	2022-10-26 08:35:53 -04:00
GMFTBY	71786b10c5	Adding the state-of-the-art contrastive search decoding methods for the codebase of generation_utils.py (#19477 ) * add: the contrastive search for generaton_utils * add: testing scripts for contrastive search under examples/text-generation * update the quality of codes * revise the docstring; make the generation_contrastive_search.py scripts; * revise the examples/pytorch/text-generation/run_generation_contrastive_search.py to the auto-APIs format * revise the necessary documents * fix: revise the docstring of generation_contrastive_search.py * Fix the code indentation * fix: revise the nits and examples in contrastive_search docstring. * fix the copyright * delete generation_contrastive_search.py * revise the logic in contrastive_search * update the intergration test and the docstring * run the tests over * add the slow decorate to the contrastive_search intergrate test * add more test * do the style, quality, consistency checks	2022-10-19 10:17:46 +01:00
amyeroberts	31ec424b3d	Add decorator to flaky test (#19674 )	2022-10-18 18:51:37 +01:00
David Yang	a23819ed6a	Clean up deprecation warnings (#19654 ) * Clean up deprecation warnings Notes: Changed some strings in tests to raw strings, which will change the literal content of the strings as they are fed into whatever machine handles them. Test cases for past in the past/past_key_values switch changed/removed due to warning of impending removal * Add PILImageResampling abstraction for PIL.Image.Resampling	2022-10-18 13:34:47 -04:00
Yifan Yang	94d7c3ba44	[Examples] make default preprocessing_num_workers=1 (#19684 ) * [Examples] make default preprocessing_num_workers=1 * [Examples] revert changes in research projects	2022-10-17 14:17:01 -04:00
Spacefish	2af36f957f	Add pillow to layoutlmv3 example requirements.txt (#19663 )	2022-10-17 08:41:57 -04:00
Duong A. Nguyen	4212bb0d60	[Re-submit] Compute true loss Flax examples (#19504 ) * Compute true loss * fixup * final * final * final * Update examples/flax/language-modeling/run_bart_dlm_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * jax.tree_map => jax.tree_util.tree_map * Compute true loss * final * fixup * final * final * Update examples/flax/language-modeling/run_bart_dlm_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * jax.tree_map => jax.tree_util.tree_map Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>	2022-10-13 11:33:36 +01:00
Sanchit Gandhi	eefcecaa35	[Examples] Fix typos in run speech recognition seq2seq (#19514 )	2022-10-12 15:33:22 +01:00
FilipposVentirozos	4ed0fa3676	Fix pytorch seq2seq qa (#19258 ) * fixed typo for SQuAD * Fixed the preprocess_validation_function function for the labels to reflect the remaining truncated instances * Rolled back the trainer_seq2seq_qa.py for UnboundLocalError: local variable 'metrics' referenced before assignment Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>	2022-10-12 08:33:44 -04:00
regisss	bb2cfd1824	Add multi-node conditions in trainer_qa.py and trainer_seq2seq.py (#19502 ) * Add multi-node conditions in trainer_qa.py and trainer_seq2seq.py * Code improvement	2022-10-11 22:48:56 -04:00
Lysandre	10100979ed	Dev version	2022-10-10 17:25:40 -04:00
Kaiyu Yang	e150c4e2fe	Fix the error message in run_t5_mlm_flax.py (#19282 )	2022-10-10 14:51:11 +01:00
wei zhao	7d5ce6802e	Fix typo in image-classification/README.md (#19424 ) Fix link typo of the following content. PyTorch version, Trainer PyTorch version, no Trainer	2022-10-10 09:16:58 -04:00
ddobokki	fa4bcd5274	edit: cast attention_mask to long in DataCollatorCTCWithPadding (#19369 ) * edit: casting attention_mask to long in DataCollatorCTCWithPadding * edit: casting attention_mask to long in DataCollatorCTCWithPadding	2022-10-07 10:05:48 -04:00
Ilaygoldman	ce2620194b	Change link of repojacking vulnerable link (#19393 ) The link to https://github.com/vasudevgupta7/bigbird is vulnerable to repojacking (it redirects to the orignial project that changed name), you should change the link to the current name of the project. if you won't change the link, an attacker can open the linked repository and attacks users that trust your links	2022-10-06 23:06:39 +02:00
Zachary Mueller	ad98642a82	Fix gather for metrics (#19360 )	2022-10-05 14:52:01 -04:00
dependabot[bot]	4c962d5e79	Bump joblib in /examples/research_projects/visual_bert (#19269 ) Bumps [joblib](https://github.com/joblib/joblib) from 0.16.0 to 1.2.0. - [Release notes](https://github.com/joblib/joblib/releases) - [Changelog](https://github.com/joblib/joblib/blob/master/CHANGES.rst) - [Commits](https://github.com/joblib/joblib/compare/0.16.0...1.2.0) --- updated-dependencies: - dependency-name: joblib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-03 23:57:50 +02:00
dependabot[bot]	c7ec0afce0	Bump joblib in /examples/research_projects/decision_transformer (#19270 ) Bumps [joblib](https://github.com/joblib/joblib) from 1.1.0 to 1.2.0. - [Release notes](https://github.com/joblib/joblib/releases) - [Changelog](https://github.com/joblib/joblib/blob/master/CHANGES.rst) - [Commits](https://github.com/joblib/joblib/compare/1.1.0...1.2.0) --- updated-dependencies: - dependency-name: joblib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-03 23:57:37 +02:00
dependabot[bot]	ca26277e33	Bump joblib from 0.16.0 to 1.2.0 in /examples/research_projects/lxmert (#19268 ) Bumps [joblib](https://github.com/joblib/joblib) from 0.16.0 to 1.2.0. - [Release notes](https://github.com/joblib/joblib/releases) - [Changelog](https://github.com/joblib/joblib/blob/master/CHANGES.rst) - [Commits](https://github.com/joblib/joblib/compare/0.16.0...1.2.0) --- updated-dependencies: - dependency-name: joblib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-10-03 23:49:35 +02:00
Divyanshu Kumar	c28d04e9e2	Update no_trainer script for summarization (#19277 ) * Update no_trainer script for summarization * removed unnecessary import * fixes notation mistake * removed: unused variable	2022-10-03 09:21:51 -04:00
Sylvain Gugger	0fc68a7e14	Fix seq2seq QA example	2022-09-28 15:45:49 -04:00
Tatsuki Okada	4a0b958d61	Fix trainer seq2seq qa.py evaluate log and ft script (#19208 ) * fix args option * fix trainer eval log * fix out of memory qa script * do isort, black, flake * fix tokenize target * take it back. * fix: comment	2022-09-28 10:55:46 -04:00
Arijit Mukherjee	ea540a5977	add wav2vec2_alignment (#16782 ) * add wav2vec2_alignment * Update alignment.py * Update examples/research_projects/wav2vec2/alignment.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Update examples/research_projects/wav2vec2/alignment.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Update examples/research_projects/wav2vec2/alignment.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Update examples/research_projects/wav2vec2/alignment.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Update README.md * fix style * fix imports * fix multithread * fix bash script * [@anton-l] Style fixes and docstrings * [@anton-l] Style fixes and docstrings * Update alignment.py fix blank id in backtrack Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: anton-l <aglozhkov@gmail.com>	2022-09-27 13:12:56 +02:00
Sylvain Gugger	c20b2c7e18	Use repo_type instead of deprecated datasets repo IDs (#19202 ) * Use repo_type instead of deprecated datasets repo IDs * Add missing one in doc	2022-09-26 09:50:48 -04:00
dependabot[bot]	98af4f9b54	Bump protobuf in /examples/research_projects/decision_transformer (#19176 ) Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.19.4 to 3.19.5. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.19.4...v3.19.5) --- updated-dependencies: - dependency-name: protobuf dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-09-26 14:55:16 +02:00
Enze	5da6afdd8d	Update run_clip.py (#19130 ) The overwrite_cache parameter is declared twice.	2022-09-23 20:48:41 +02:00
dependabot[bot]	74a3ea4737	Bump oauthlib in /examples/research_projects/decision_transformer (#19080 ) Bumps [oauthlib](https://github.com/oauthlib/oauthlib) from 3.2.0 to 3.2.1. - [Release notes](https://github.com/oauthlib/oauthlib/releases) - [Changelog](https://github.com/oauthlib/oauthlib/blob/master/CHANGELOG.rst) - [Commits](https://github.com/oauthlib/oauthlib/compare/v3.2.0...v3.2.1) --- updated-dependencies: - dependency-name: oauthlib dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-09-22 17:01:40 +02:00
Matt	83dc6377d0	Reduce LR for TF MLM example test (#19156 )	2022-09-22 08:51:27 -04:00
Leandro von Werra	ef6741fe65	Fix GLUE MNLI when using `max_eval_samples` (#18722 )	2022-09-21 09:33:22 +02:00
Santiago Castro	06f341de4f	Add a missing space in a script arg documentation (#19113 )	2022-09-20 21:43:32 +02:00
Ryan X	36e356caa4	Fix: update ltp word segmentation call in mlm_wwm (#19047 ) * Fix: update ltp word segmentation call in mlm_wwm * Fix: update ltp word segmentation call in mlm_wwm * Fix: update ltp word segmentation call in mlm_wwm	2022-09-20 09:20:38 -04:00
dependabot[bot]	7d0486c106	Bump mako in /examples/research_projects/decision_transformer (#19077 ) Bumps [mako](https://github.com/sqlalchemy/mako) from 1.2.0 to 1.2.2. - [Release notes](https://github.com/sqlalchemy/mako/releases) - [Changelog](https://github.com/sqlalchemy/mako/blob/main/CHANGES) - [Commits](https://github.com/sqlalchemy/mako/commits) --- updated-dependencies: - dependency-name: mako dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-09-16 22:15:02 +02:00
Lysandre	16913b3c92	Dev version	2022-09-14 14:58:20 -04:00
Rahul A R	00fc9217d1	Fixed bug which caused overwrite_cache to always be True (#19000 ) * fixed bug which caused overwrite_cache to always be True (#18967). * reformatting changes	2022-09-13 11:29:48 -04:00
Rafał Jankowski	85125fcffd	Neptune.ai integration improvements (#18934 ) * NeptuneCallback improvements * After review suggestions and deduplication of initial run * Added volatile checkpoints support due to missing post-rebase commit * Update README per review comments - Remove list formatting - Correct Neptune docs link Co-authored-by: Sabine <sabine.nyholm@neptune.ai>	2022-09-09 11:37:34 -04:00
Sanchit Gandhi	e6f221c8d4	[JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* (#18361 ) * [JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* * fix double tree_util	2022-09-09 15:18:56 +02:00
Nicholas Broad	4f299b2446	Accelerator end training (#18910 ) * add accelerator.end_training() Some trackers need this to end their runs. * fixup and quality * add space * add space again ?!?	2022-09-07 07:46:26 -04:00
arun99481	3b19c0317b	updating gather function with gather_for_metrics in run_wav2vec2_pretraining (#18877 ) Co-authored-by: Arun Rajaram <arunrajaram@Aruns-MacBook-Pro.local>	2022-09-06 07:36:37 -04:00
Sylvain Gugger	c61f116b63	Tie weights after preparing the model in run_clm (#18855 )	2022-09-01 12:06:56 -04:00
Rahul A R	e9442440fc	streamlining 'checkpointing_steps' parsing (#18755 )	2022-08-25 11:00:38 -04:00
Rahul A R	c55d6e4e10	examples/run_summarization_no_trainer: fixed incorrect param to hasattr (#18720 ) * fixed incorrect param to hasattr * simplified condition checks * code cleanup	2022-08-24 12:12:42 -04:00
dependabot[bot]	e49c71fc4c	Bump nbconvert from 6.3.0 to 6.5.1 in /examples/research_projects/lxmert (#18742 ) Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.3.0 to 6.5.1. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](https://github.com/jupyter/nbconvert/compare/6.3.0...6.5.1) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-24 06:12:56 -04:00
dependabot[bot]	5b24949669	Bump nbconvert in /examples/research_projects/visual_bert (#18741 ) Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.3.0 to 6.5.1. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](https://github.com/jupyter/nbconvert/compare/6.3.0...6.5.1) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-24 06:12:48 -04:00
Atharva Ingle	d90a36d192	remove check for main process for trackers initialization (#18706 )	2022-08-22 11:16:27 -04:00
Atharva Ingle	e54a1b49aa	`model.tie_weights()` should be applied after `accelerator.prepare()` (#18676 ) * `model.tie_weights()` should be applied after `accelerator.prepare` Weight tying should be done after the model has been moved to XLA device as mentioned on PyTorch/XLA Troubleshooting guide [here](https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md#xla-tensor-quirks) * format code	2022-08-18 13:46:57 -04:00
Loubna Ben Allal	bbbb453e58	Add an examples folder for code downstream tasks (#18679 ) * add examples subfolder * mention examples in codeparrot readme * use Trainer optimizer and scheduler type and add output_dir as argument * add example of text-to-python and python-to-text models * mention the downstream examples in the readme * fix typo	2022-08-18 18:24:24 +02:00
Zachary Mueller	358fc18613	Add evaluate to examples requirements (#18666 )	2022-08-18 10:57:39 -04:00
Stefan Schweter	358478e729	Examples: add Bloom support for token classification (#18632 ) * examples: add Bloom support for token classification (FLAX, PyTorch and TensorFlow) * examples: remove support for Bloom in token classication (FLAX and TensorFlow currently have no support for it)	2022-08-17 09:50:57 +02:00
zhoutang776	25e651a2de	Update run_translation_no_trainer.py (#18637 ) * Update run_translation_no_trainer.py found an error in selecting `no_decay` parameters and some small modifications when the user continues to train from a checkpoint * fixs `no_decay` and `resume_step` issue 1. change `no_decay` list 2. if use continue to train their model from provided checkpoint, the `resume_step` will not be initialized properly if `args.gradient_accumulation_steps != 1`	2022-08-16 13:25:57 -04:00
Karim Foda	d6eeb87170	Flax Remat for LongT5 (#17994 ) * [Flax] Add remat (gradient checkpointing) * fix variable naming in test * flip: checkpoint using a method * fix naming * fix class naming * apply PVP's suggestions from code review * add gradient_checkpointing to examples * Add gradient_checkpointing to run_mlm_flax * Add remat to longt5 * Add gradient checkpointing test longt5 * Fix args errors * Fix remaining tests * Make fixup & quality fixes * replace kwargs * remove unecessary kwargs * Make fixup changes * revert long_t5_flax changes * Remove return_dict and copy to LongT5 * Remove test_gradient_checkpointing Co-authored-by: sanchit-gandhi <sanchit@huggingface.co>	2022-08-14 16:27:13 +01:00
dependabot[bot]	05d3a43c59	Bump nbconvert in /examples/research_projects/visual_bert (#18566 ) Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.0.1 to 6.3.0. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](https://github.com/jupyter/nbconvert/compare/6.0.1...6.3.0) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-11 10:47:31 -04:00
dependabot[bot]	713ab6fde5	Bump nbconvert from 6.0.1 to 6.3.0 in /examples/research_projects/lxmert (#18565 ) Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.0.1 to 6.3.0. - [Release notes](https://github.com/jupyter/nbconvert/releases) - [Commits](https://github.com/jupyter/nbconvert/compare/6.0.1...6.3.0) --- updated-dependencies: - dependency-name: nbconvert dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-11 10:47:19 -04:00
Matt	6eb51450fa	TF Examples Rewrite (#18451 ) * Finished QA example * Dodge a merge conflict * Update text classification and LM examples * Update NER example * New Keras metrics WIP, fix NER example * Update NER example * Update MC, summarization and translation examples * Add XLA warnings when shapes are variable * Make sure batch_size is consistently scaled by num_replicas * Add PushToHubCallback to all models * Add docs links for KerasMetricCallback * Add docs links for prepare_tf_dataset and jit_compile * Correct inferred model names * Don't assume the dataset has 'lang' * Don't assume the dataset has 'lang' * Write metrics in text classification * Add 'framework' to TrainingArguments and TFTrainingArguments * Export metrics in all examples and add tests * Fix training args for Flax * Update command line args for translation test * make fixup * Fix accidentally running other tests in fp16 * Remove do_train/do_eval from run_clm.py * Remove do_train/do_eval from run_mlm.py * Add tensorflow tests to circleci * Fix circleci * Update examples/tensorflow/language-modeling/run_mlm.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/test_tensorflow_examples.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/translation/run_translation.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update examples/tensorflow/token-classification/run_ner.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Fix save path for tests * Fix some model card kwargs * Explain the magical -1000 * Actually enable tests this time * Skip text classification PR until we fix shape inference * make fixup Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>	2022-08-10 16:49:51 +01:00
Rasmus Arpe Fogh Jensen	a765b68aa6	Update no_trainer.py scripts to include accelerate gradient accumulation wrapper (#18473 ) * Added accelerate gradient accumulation wrapper to run_image_classification_no_trainer.py example script * make fixup changes * PR comments * changed input to Acceletor based on PR comment, ran make fixup * Added comment explaining the sync_gradients statement * Fixed lr scheduler max steps * Changed run_clm_no_trainer.py script to use accelerate gradient accum wrapper * Fixed all scripts except wav2vec2 pretraining to use accelerate gradient accum wrapper * Added accelerate gradient accum wrapper for wav2vec2_pretraining_no_trainer.py script * make fixup and lr_scheduler step inserted back into run_qa_beam_search_no_trainer.py * removed changes to run_wav2vec2_pretraining_no_trainer.py script and fixed using wrong constant in qa_beam_search_no_trainer.py script	2022-08-08 15:52:47 -04:00
Sylvain Gugger	70b0d4e193	Fix compatibility with 1.12 (#17925 ) * Fix compatibility with 1.12 * Remove pin from examples requirements * Update torch scatter version * Fix compatibility with 1.12 * Remove pin from examples requirements * Update torch scatter version * fix torch.onnx.symbolic_opset12 import * Reject bad version Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2022-08-08 09:53:08 -04:00
regisss	88a0ce57bb	Add seed setting to image classification example (#18519 )	2022-08-08 08:08:11 -04:00
Julien Chaumond	9129fd0377	`transformers-cli login` => `huggingface-cli login` (#18490 ) * zero chance anyone's using that constant no? * `transformers-cli login` => `huggingface-cli login` * `transformers-cli repo create` => `huggingface-cli repo create` * `make style`	2022-08-06 09:42:55 +02:00
Julien Chaumond	8d1f9039d0	Just re-reading the whole doc every couple of months 😬 (#18489 ) * Delete valohai.yaml * NLP => ML * typo * website supports https * datasets * 60k + modalities * unrelated link fixing for accelerate * Ok those links were actually broken * Fix link * Make `AutoTokenizer` auto-link * wording tweak * add at least one non-nlp task	2022-08-06 09:38:55 +02:00
Kian Sierra McGettigan	0bf1e1aca4	Update no trainer examples for QA and Semantic Segmentation (#18474 ) * swag_no_trainer updated for with gather_metrics * Removed unused variable samples_seen * updated examples with gather_for_metrics	2022-08-04 13:22:19 -04:00
Kian Sierra McGettigan	330247ede2	Update no trainer scripts for multiple-choice (#18468 ) * swag_no_trainer updated for with gather_metrics * Removed unused variable samples_seen	2022-08-04 07:29:32 -04:00
LSinev	02b176c4ce	Fix torch version comparisons (#18460 ) Comparisons like version.parse(torch.__version__) > version.parse("1.6") are True for torch==1.6.0+cu101 or torch==1.6.0+cpu version.parse(version.parse(torch.__version__).base_version) are preferred (and available in pytorch_utils.py	2022-08-03 13:37:18 -04:00
Ritik Nandwal	3db4378bd7	Update no trainer scripts for language modeling and image classification examples (#18443 ) * Update no_trainer script for image-classification * Update no_trainer scripts for language-modeling examples * Remove unused variable * Removing truncation from losses array for language modeling examples	2022-08-03 08:33:18 -04:00
Yih-Dar	5546fb61ab	fix run_clip README (#18332 ) Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>	2022-08-02 19:14:46 +02:00
Duong A. Nguyen	3909d7f139	Add Flax BART pretraining script (#18297 ) * add bart pretraining flax script * fixup * add bart pretraining flax script * add BART to README * add BART to README * add BART to README * add BART to README * add BART to README * add bos eos document * Update README.md * Update README.md * Update examples/flax/language-modeling/run_bart_dlm_flax.py Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com> * final * final * final * remove use_auth_token ing from_config Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>	2022-08-01 12:06:30 -04:00
Sylvain Gugger	941d233153	Fix ROUGE add example check and update README (#18398 ) * Fix ROUGE add example check and update README * Stay consistent in values	2022-08-01 11:14:49 -04:00
Ogundepo Odunayo	679d68a11b	Correct the spelling of bleu metric (#18375 )	2022-08-01 07:51:27 -04:00
atturaioe	1f84399171	Migrate metric to Evaluate in Pytorch examples (#18369 ) * Migrate metric to Evaluate in pytorch examples * Remove unused imports	2022-08-01 07:40:25 -04:00
dependabot[bot]	25ec12eaf7	Bump mistune from 0.8.4 to 2.0.3 in /examples/research_projects/lxmert (#18370 ) Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3. - [Release notes](https://github.com/lepture/mistune/releases) - [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst) - [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3) --- updated-dependencies: - dependency-name: mistune dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-01 04:46:57 -04:00
dependabot[bot]	a7360385f4	Bump mistune in /examples/research_projects/visual_bert (#18371 ) Bumps [mistune](https://github.com/lepture/mistune) from 0.8.4 to 2.0.3. - [Release notes](https://github.com/lepture/mistune/releases) - [Changelog](https://github.com/lepture/mistune/blob/master/docs/changes.rst) - [Commits](https://github.com/lepture/mistune/compare/v0.8.4...v2.0.3) --- updated-dependencies: - dependency-name: mistune dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2022-08-01 04:46:31 -04:00
Sylvain Gugger	986526a0e4	Replace `as_target` context managers by direct calls (#18325 ) * Preliminary work on tokenizers * Quality + fix tests * Treat processors * Fix pad * Remove all uses of in tests, docs and examples * Replace all as_target_tokenizer * Fix tests * Fix quality * Update examples/flax/image-captioning/run_image_captioning_flax.py Co-authored-by: amyeroberts <amy@huggingface.co> * Style Co-authored-by: amyeroberts <amy@huggingface.co>	2022-07-29 08:09:09 -04:00
Vijay S Kalmath	da503ea02f	Migrate metrics used in flax examples to Evaluate (#18348 ) Currently, tensorflow examples use the `load_metric` function from Datasets library, commit migrates function call to `load` function from Evaluate library.	2022-07-28 15:06:23 -04:00

... 4 5 6 7 8 ...

2613 Commits