riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 10:12:23 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity
5739726fcc
transformers/tests
History
Connor Henderson 5739726fcc
fix: Text splitting in the BasicTokenizer (#22280) 
* fix: Apostraphe splitting in the BasicTokenizer for CLIPTokenizer

* account for apostrophe at start of new word

* remove _run_split_on_punc, use re.findall instead

* remove debugging, make style and quality

* use pattern and punc splitting, repo-consistency will fail

* remove commented out debugging

* adds bool args to BasicTokenizer, remove pattern

* do_split_on_punc default True

* clean stray comments and line breaks

* rebase, repo-consistency

* update to just do punctuation split

* add unicode normalizing back

* remove redundant line
2023-07-11 11:07:58 -04:00
..
benchmark [Test refactor 1/5] Per-folder tests reorganization (#15725) 2022-02-23 15:46:28 -05:00
bettertransformer Add methods to PreTrainedModel to use PyTorch's BetterTransformer (#21259) 2023-04-27 11:03:42 +02:00
bnb [gpt2-int8] Add gpt2-xl int8 test (#24543) 2023-06-28 18:02:13 +02:00
deepspeed accelerate deepspeed and gradient accumulation integrate (#23236) 2023-05-31 15:16:22 +05:30
extended [tests] switch to torchrun (#22712) 2023-04-12 08:25:45 -07:00
fixtures [WIP] add SpeechT5 model (#18922) 2023-02-03 12:43:46 -05:00
generation Fix TypeError: Object of type int64 is not JSON serializable (#24340) 2023-06-27 12:15:49 +01:00
models fix: Text splitting in the BasicTokenizer (#22280) 2023-07-11 11:07:58 -04:00
onnx Fix TypeError: Object of type int64 is not JSON serializable (#24340) 2023-06-27 12:15:49 +01:00
optimization Make schedulers picklable by making lr_lambda fns global (#21768) 2023-03-02 12:08:43 -05:00
pipelines [pipeline] Fix str device issue (#24396) 2023-06-26 13:58:36 +02:00
repo_utils Fix expected value in tests of the test fetcher (#24077) 2023-06-07 11:38:56 -04:00
sagemaker Avoid invalid escape sequences, use raw strings (#22936) 2023-04-25 09:17:56 -04:00
tokenization Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
tools Add support for for loops in python interpreter (#24429) 2023-06-26 09:58:14 -04:00
trainer Fix TypeError: Object of type int64 is not JSON serializable (#24340) 2023-06-27 12:15:49 +01:00
utils Make (TF) CI faster (test only a subset of model classes) (#24592) 2023-06-30 16:54:54 +02:00
__init__.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_backbone_common.py Add TimmBackbone model (#22619) 2023-06-06 17:11:30 +01:00
test_configuration_common.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_configuration_utils.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_feature_extraction_common.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_feature_extraction_utils.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_image_processing_common.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_image_processing_utils.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_image_transforms.py Bug fix - flip_channel_order for channels first images (#23701) 2023-05-31 17:12:27 +01:00
test_modeling_common.py Make PT/Flax tests could be run on GPU (#24557) 2023-06-28 20:11:01 +02:00
test_modeling_flax_common.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_modeling_flax_utils.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_modeling_tf_common.py Speed up TF tests by reducing hidden layer counts (#24595) 2023-06-30 16:30:33 +01:00
test_modeling_tf_utils.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00
test_modeling_utils.py Show a warning for missing attention masks when pad_token_id is not None (#24510) 2023-06-30 08:19:39 -04:00
test_pipeline_mixin.py Update tiny models for pipeline testing. (#24364) 2023-06-20 14:43:10 +02:00
test_sequence_feature_extraction_common.py Apply ruff flake8-comprehensions (#21694) 2023-02-22 09:14:54 +01:00
test_tokenization_common.py Fix TypeError: Object of type int64 is not JSON serializable (#24340) 2023-06-27 12:15:49 +01:00
test_tokenization_utils.py Split common test from core tests (#24284) 2023-06-15 07:30:24 -04:00