riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-05 05:40:05 +06:00
Code Issues Actions 27 Packages Projects Releases Wiki Activity
15,901 Commits 861 Branches 225 Tags 2.1 GiB
57c965a8f1
Commit Graph

51 Commits

Author SHA1 Message Date
amyeroberts
57c965a8f1
Remove deprecated logic and warnings (#30743) 
* Remove deprecated logic and warnings

* Add back some code that seems to be important...

* Let's just add all he nllb stuff back; removing it is a bit more involved

* Remove kwargs

* Remove more kwargs
 2024-05-17 12:15:59 +01:00
Hilco van der Wilk
b6404866cd
Update legacy Repository usage in various example files (#29085) 
* Update legacy Repository usage in `examples/pytorch/text-classification/run_glue_no_trainer.py`

Marked for deprecation here https://huggingface.co/docs/huggingface_hub/guides/upload#legacy-upload-files-with-git-lfs

* Fix import order

* Replace all example usage of deprecated Repository

* Fix remaining repo call and rename args variable

* Revert removing creation of gitignore files and don't change research examples
 2024-03-12 13:20:49 +00:00
zspo
d98591a12b
[docs] fix some bugs about parameter description (#28806) 
Co-authored-by: p_spozzhang <p_spozzhang@tencent.com>
 2024-02-01 16:59:29 +00:00
Klaus Hipp
39fa400969
Fix input data file extension in examples (#28741) 2024-01-29 10:06:31 +00:00
V.Prasanna kumar
ffbcfc0166
Broken links fixed related to datasets docs (#27569) 
fixed the broken links belogs to dataset library of transformers
 2023-11-17 13:44:09 -08:00
Lucain
66b088faf0
Provide alternative when warning on use_auth_token (#27105) 2023-10-27 14:32:54 +02:00
Tom Aarsen
40ea9ab2a1
Add many missing spaces in adjacent strings (#26751) 
Add missing spaces in adjacent strings
 2023-10-12 10:28:40 +02:00
Roy Hvaara
fc63914399
[JAX] Replace uses of jnp.array in types with jnp.ndarray. (#26703) 
`jnp.array` is a function, not a type:
https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.array.html
so it never makes sense to use `jnp.array` in a type annotation. Presumably the intent was to write `jnp.ndarray` aka `jax.Array`.

Co-authored-by: Peter Hawkins <phawkins@google.com>
 2023-10-10 21:35:16 +02:00
Phuc Van Phan
910faa3e1f
feat: adding num_proc to load_dataset (#26326) 
* feat: adding num_proc to load_dataset

* feat: add add_num_proc for run_mlm_flax

* feat: add num_proc for bart and t5

* chorse: remove
 2023-09-22 19:22:47 +02:00
Phuc Van Phan
8b5da9fc6e
refactor: change default block_size in block size > max position embeddings (#26069) 
* refactor: change default block_size when not initialize

* reformat: add the min of block size
 2023-09-18 16:47:57 +01:00
Phuc Van Phan
5af2c62696
docs: add space to docs (#26067) 
* docs: add space to docs

* docs: remove reduntant space
 2023-09-11 22:03:26 +01:00
Phuc Van Phan
9cebae64ad
docs: update link huggingface map (#26077) 2023-09-11 12:57:04 +01:00
Jackmin801
145109382a
Allow trust_remote_code in example scripts (#25248) 
* pytorch examples

* pytorch mim no trainer

* cookiecutter

* flax examples

* missed line in pytorch run_glue

* tensorflow examples

* tensorflow run_clip

* tensorflow run_mlm

* tensorflow run_ner

* tensorflow run_clm

* pytorch example from_configs

* pytorch no trainer examples

* Revert "tensorflow run_clip"

This reverts commit 261f86ac1f.

* fix: duplicated argument
 2023-08-07 16:32:25 +02:00
Yih-Dar
149cb0cce2
Add token arugment in example scripts (#25172) 
* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-02 11:17:31 +02:00
Yih-Dar
d53b8ad780
Update use_auth_token -> token in example scripts (#25167) 
* pytorch examples

* tensorflow examples

* flax examples

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-07-28 15:33:45 +02:00
Lucain
6232c380f2
Fix .push_to_hub and cleanup get_full_repo_name usage (#25120) 
* Fix .push_to_hub and cleanup get_full_repo_name usage

* Do not rely on Python bool conversion magic

* request changes
 2023-07-28 11:40:08 +02:00
Bauke Brenninkmeijer
fc9e387dc0
Replacement of 20 asserts with exceptions (#24757) 
* initial replacements of asserts with errors/exceptions

* replace assert with exception in generation, align and bart

* reset formatting change

* reset another formatting issue

* Apply suggestion

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* don't touch this file

* change to 'is not False'

* fix type

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2023-07-12 07:45:09 -04:00
Aaron Gokaslan
5e8c8eb5ba
Apply ruff flake8-comprehensions (#21694) 2023-02-22 09:14:54 +01:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting (#21480) 
* Result of black 23.1

* Update target to Python 3.7

* Switch flake8 to ruff

* Configure isort

* Configure isort

* Apply isort with line limit

* Put the right black version

* adapt black in check copies

* Fix copies
 2023-02-06 18:10:56 -05:00
Sylvain Gugger
05e72aa0c4
Adapt repository creation to latest hf_hub (#21158) 
* Adapt repository creation to latest hf_hub

* Update all examples

* Fix other tests, add Flax examples

* Address review comments
 2023-01-18 11:14:00 -05:00
Sanchit Gandhi
e6f221c8d4
[JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* (#18361) 
* [JAX] Replace all jax.tree_* calls with jax.tree_util.tree_*

* fix double tree_util
 2022-09-09 15:18:56 +02:00
Julien Chaumond
9129fd0377
transformers-cli login => huggingface-cli login (#18490) 
* zero chance anyone's using that constant no?

* `transformers-cli login` => `huggingface-cli login`

* `transformers-cli repo create` => `huggingface-cli repo create`

* `make style`
 2022-08-06 09:42:55 +02:00
Sanchit Gandhi
7490a97cac
[Flax] Fix incomplete batches in example scripts (#17863) 
* [Flax] Fix incomplete batches in example scripts

* fix dataloader batching

* convert jnp batch idxs to np array

* add missing `pad_shard_unpad` to final prediction generate step

* only `pad_shard_unpad` at inference time

* merge conflicts

* remove incomplete batch step from eval

* fix run_qa.py

* add `pad_shard_unpad` to run_flax_ner.py

* add `pad_shard_unpad` to run_flax_glue.py

* add `pad_shard_unpad` to run_image_classification.py

* make style

* fix mlm flax eval batches

* remove redundant imports
 2022-07-27 15:50:47 +01:00
Duong A. Nguyen
170fcaa604
Generalize decay_mask_fn to apply mask to all LayerNorm params (#18273) 
* generalize decay_mask_fn to find all layernorm params

* fixup

* generalising decay_mask_fn
 2022-07-27 12:23:57 +01:00
Duong A. Nguyen
4bea6584e3
Remove use_auth_token from the from_config method (#18192) 
* remove use_auth_token from from_config

* restore use_auth_token from_pretrained run_t5_mlm_flax
 2022-07-19 08:13:20 +02:00
Sylvain Gugger
3cab90279f
Add examples telemetry (#17552) 
* Add examples telemetry

* Alternative approach

* Add to all other examples

* Add to templates as well

* Put framework separately

* Same for TensorFlow
 2022-06-07 11:57:52 -04:00
Sylvain Gugger
afe5d42d8d
Black preview (#17217) 
* Black preview

* Fixup too!

* Fix check copies

* Use the same version as the CI

* Bump black
 2022-05-12 16:25:55 -04:00
Karim Foda
24a85cca61
Add use_auth to load_datasets for private datasets to PT and TF examples (#16521) 
* fix formatting and remove use_auth

* Add use_auth_token to Flax examples
 2022-04-04 10:27:45 -04:00
Stas Bekman
a73281e3e4
[examples] max samples can't be bigger than the len of dataset (#16501) 
* [examples] max samples can't be bigger than then len of dataset

* do tf and flax
 2022-03-30 12:33:16 -07:00
Sylvain Gugger
4975002df5
Reorganize file utils (#16264) 
* Split file_utils in several submodules

* Fixes

* Add back more objects

* More fixes

* Who exactly decided to import that from there?

* Second suggestion to code with code review

* Revert wront move

* Fix imports

* Adapt all imports

* Adapt all imports everywhere

* Revert this import, will fix in a separate commit
 2022-03-23 10:26:33 -04:00
Benjamin Minixhofer
2a606f9974
Make data shuffling in run_clm_flax.py respect global seed (#13410) 
* use jax and jnp instead of numpy in data_loader

* return batches as np.ndarray
 2021-12-14 11:04:43 +01:00
Suraj Patil
6a025487a6
[Flax examples] remove dependancy on pytorch training args (#14636) 
* use custom training arguments

* update tests
 2021-12-12 09:19:12 +05:30
Julien Chaumond
6cdc3a7844
[urls to hub] Replace outdated model tags with their now-canonical pipeline types (#14617) 
* Replace outdated model tags with their now-canonical pipeline types

* spam the CI till it's green
 2021-12-06 04:35:01 -05:00
Suraj Patil
c5bd732ac6
Add Flax example tests (#14599) 
* add test for glue

* add tests for clm

* fix clm test

* add summrization tests

* more tests

* fix few tests

* add test for t5 mlm

* fix t5 mlm test

* fix tests for multi device

* cleanup

* ci job

* fix metric file name

* make t5 more robust
 2021-12-06 10:48:58 +05:30
Nicholas Broad
69e16abf98
Switch from using sum for flattening lists of lists in group_texts (#14472) 
* remove sum for list flattening

* change to chain(*)

* make chain object a list

* delete empty lines

per sgugger's suggestions

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Nicholas Broad <nicholas@nmbroad.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-11-22 16:17:26 -05:00
Suraj Patil
7db2a79b38
[examples/flax] use Repository API for push_to_hub (#13672) 
* use Repository for push_to_hub

* update readme

* update other flax scripts

* update readme

* update qa example

* fix push_to_hub call

* fix typo

* fix more typos

* update readme

* use abosolute path to get repo name

* fix glue script
 2021-09-30 16:38:07 +05:30
Stefan Schweter
4046e66e40
examples: only use keep_linebreaks when reading TXT files (#13320) 
* examples: only use keep_linebreaks when reading TXT files for all CLM examples

* examples: only use keep_linebreaks when reading TXT files for all CLM examples

* examples: only use keep_linebreaks when reading TXT files for all CLM examples
 2021-08-28 16:22:29 +02:00
Stefan Schweter
319d840b46
examples: add keep_linebreaks option to CLM examples (#13150) 
* examples: add keep_linebreaks option to text dataset loader for all CLM examples

* examples: introduce new keep_linebreaks option as data argument in CLM examples
 2021-08-27 11:35:45 +02:00
Patrick von Platen
13a9c9a354
[Flax] Refactor gpt2 & bert example docs (#13024) 
* fix_torch_device_generate_test

* remove @

* improve docs for clm

* speed-ups

* correct t5 example as well

* push final touches

* Update examples/flax/language-modeling/README.md

* correct docs for mlm

* Update examples/flax/language-modeling/README.md

Co-authored-by: Patrick von Platen <patrick@huggingface.co>
 2021-08-09 13:37:50 +02:00
Stefan Schweter
d3c3e722d6
[FLAX] Minor fixes in CLM example (#12914) 
* readme: fix retrieval of vocab size for flax clm example

* examples: fix flax clm example when using training/evaluation files
 2021-07-27 19:48:04 +05:30
fgaim
66197adc98
Flax MLM: Allow validation split when loading dataset from local file (#12689) 
* Allow validation split when loading dataset from local file

* Flax clm & t5, enable validation split for datasets loaded from local file
 2021-07-20 13:38:25 +02:00
Patrick von Platen
deecdd4939
[Flax] Fix cur step flax examples (#12608) 
* fix_torch_device_generate_test

* remove @

* fix save problem
 2021-07-09 13:51:28 +01:00
Sylvain Gugger
6f1adc4334
Fix group_lengths for short datasets (#12558) 2021-07-08 07:23:41 -04:00
Suraj Patil
2d42915abe
[examples/flax] add adafactor optimizer (#12544) 
* add adafactor

* Update examples/flax/language-modeling/run_mlm_flax.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
 2021-07-07 11:50:30 +05:30
Patrick von Platen
208df208bf
[Flax] Adapt examples to be able to use eval_steps and save_steps (#12543) 
* fix_torch_device_generate_test

* remove @

* up

* up

* correct

* upload

Co-authored-by: Patrick von Platen <patrick@huggingface.co>
 2021-07-06 19:41:51 +01:00
Patrick von Platen
d0f7508abe
[Flax] Correct logging steps flax (#12515) 
* fix_torch_device_generate_test

* remove @

* push
 2021-07-05 18:21:00 +01:00
Patrick von Platen
bb4ac2b5a8
[Flax] Correct flax training scripts (#12514) 
* fix_torch_device_generate_test

* remove @

* add logging steps

* correct training scripts

* correct readme

* correct
 2021-07-05 18:14:50 +01:00
Patrick von Platen
813328682e
[Flax] Example scripts - correct weight decay (#12409) 
* fix_torch_device_generate_test

* remove @

* finish

* finish

* correct style
 2021-06-29 12:01:08 +01:00
Patrick von Platen
2d70c91206
[Flax] Adapt flax examples to include push_to_hub (#12391) 
* fix_torch_device_generate_test

* remove @

* finish

* correct summary writer

* correct push to hub

* fix indent

* finish

* finish

* finish

* finish

* finish

Co-authored-by: Patrick von Platen <patrick@huggingface.co>
 2021-06-28 19:23:35 +01:00
Stas Bekman
4a872caef4
remove extra white space from log format (#12360) 2021-06-25 13:20:14 -07:00