riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity
19a8a3036d
transformers/tests/models
History
Matt 19a8a3036d
Add magic method to our TF models to convert datasets with column inference (#17160) 
* Add method to call to_tf_dataset() with column inference

* Add test for dataset creation

* Add a default arg for data collator

* Fix test

* Fix call with non-dev version of datasets

* Test correct column removal too

* make fixup

* More tests to make sure we remove unwanted columns

* Fix test to avoid predicting on unbuilt models

* Fix test to avoid predicting on unbuilt models

* Fix test to remove unwanted head mask columns from inputs

* Stop pushing your debug breakpoints to the main repo of the $2bn company you work for

* Skip the test in convnext because no grouped conv support

* Drop bools from the dataset dict

* Make style

* Skip the training test for models whose input dicts don't give us labels

* Skip transformerXL in the test because it doesn't return a simple loss

* Skip TFTapas because of some odd NaN losses

* make style

* make fixup

* Add docstring

* fixup

* Update src/transformers/modeling_tf_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/modeling_tf_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/modeling_tf_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/modeling_tf_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/modeling_tf_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Remove breakpoint from tests

* Fix assert, add requires_backends

* Protect tokenizer import with if TYPE_CHECKING

* make fixup

* Add noqa, more fixup

* More rearranging for ~* aesthetics *~

* Adding defaults for shuffle and batch_size to match to_tf_dataset()

* Update src/transformers/modeling_tf_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2022-06-06 15:53:49 +01:00
..
albert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
auto Automatically sort auto mappings (#17250) 2022-05-16 13:24:20 -04:00
bart Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
barthez Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
bartpho Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
beit Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
bert Black preview (#17217) 2022-05-12 16:25:55 -04:00
bert_generation Black preview (#17217) 2022-05-12 16:25:55 -04:00
bert_japanese Black preview (#17217) 2022-05-12 16:25:55 -04:00
bertweet Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
big_bird Black preview (#17217) 2022-05-12 16:25:55 -04:00
bigbird_pegasus Black preview (#17217) 2022-05-12 16:25:55 -04:00
blenderbot Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
blenderbot_small Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
bort Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
byt5 Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
camembert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
canine Black preview (#17217) 2022-05-12 16:25:55 -04:00
clip Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
convbert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
convnext Add magic method to our TF models to convert datasets with column inference (#17160) 2022-06-06 15:53:49 +01:00
cpm Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
ctrl Fix CTRL tests (#17508) 2022-06-01 16:27:23 +02:00
cvt Add CvT (#17299) 2022-05-18 17:47:18 +02:00
data2vec Black preview (#17217) 2022-05-12 16:25:55 -04:00
deberta Black preview (#17217) 2022-05-12 16:25:55 -04:00
deberta_v2 Add DebertaV2ForMultipleChoice (#17135) 2022-05-10 16:21:44 -04:00
decision_transformer Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
deit Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
detr Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
distilbert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
dit Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
dpr Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
dpt Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
electra Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
encoder_decoder Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
flaubert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
flava Fix FlavaForPreTrainingIntegrationTest CI test (#17232) 2022-05-16 21:14:25 +02:00
fnet Black preview (#17217) 2022-05-12 16:25:55 -04:00
fsmt Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
funnel Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
glpn Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
gpt_neo Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
gpt_neox [WIP] Adding GPT-NeoX-20B (#16659) 2022-05-24 09:31:10 -04:00
gpt2 TF: GPT-2 generation supports left-padding (#17426) 2022-05-31 14:06:44 +01:00
gptj Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
herbert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
hubert Black preview (#17217) 2022-05-12 16:25:55 -04:00
ibert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
imagegpt Enabling imageGPT auto feature extractor. (#16871) 2022-05-24 12:30:46 +02:00
layoutlm Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
layoutlmv2 Add LayoutLMv3 (#17060) 2022-05-24 09:53:45 +02:00
layoutlmv3 Add LayoutLMv3 (#17060) 2022-05-24 09:53:45 +02:00
layoutxlm Fix LayoutXLMProcessorTest (#17506) 2022-06-01 16:26:37 +02:00
led Black preview (#17217) 2022-05-12 16:25:55 -04:00
levit fix integration test levit (#17555) 2022-06-06 13:47:32 +02:00
longformer Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
luke Debug LukeForMaskedLM (#17499) 2022-06-01 10:03:06 -04:00
lxmert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
m2m_100 Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
marian Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
maskformer Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
mbart Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
mbart50 Black preview (#17217) 2022-05-12 16:25:55 -04:00
megatron_bert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
megatron_gpt2 Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
mluke Black preview (#17217) 2022-05-12 16:25:55 -04:00
mobilebert Black preview (#17217) 2022-05-12 16:25:55 -04:00
mpnet Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
mt5 Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
nystromformer Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
openai Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
opt Fix all offload and MP tests (#17533) 2022-06-03 09:59:13 -04:00
pegasus Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
perceiver Black preview (#17217) 2022-05-12 16:25:55 -04:00
phobert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
plbart Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
poolformer Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
prophetnet Black preview (#17217) 2022-05-12 16:25:55 -04:00
qdqbert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
rag Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
realm Black preview (#17217) 2022-05-12 16:25:55 -04:00
reformer Black preview (#17217) 2022-05-12 16:25:55 -04:00
regnet Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
rembert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
resnet Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
retribert fix retribert's test_torch_encode_plus_sent_to_model (#17231) 2022-05-17 14:33:13 +02:00
roberta Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
roformer Skip RoFormer ONNX test if rjieba not installed (#16981) 2022-05-04 10:04:10 +02:00
segformer Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
sew Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
sew_d Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
speech_encoder_decoder Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
speech_to_text Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
speech_to_text_2 Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
splinter Add support for pretraining recurring span selection to Splinter (#17247) 2022-05-17 23:42:14 +02:00
squeezebert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
swin Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
t5 Fix all offload and MP tests (#17533) 2022-06-03 09:59:13 -04:00
tapas Add magic method to our TF models to convert datasets with column inference (#17160) 2022-06-06 15:53:49 +01:00
tapex Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
trajectory_transformer Add trajectory transformer (#17141) 2022-05-17 19:07:43 -04:00
transfo_xl Add magic method to our TF models to convert datasets with column inference (#17160) 2022-06-06 15:53:49 +01:00
trocr Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
unispeech Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
unispeech_sat Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
van Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
vilt Black preview (#17217) 2022-05-12 16:25:55 -04:00
vision_encoder_decoder Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
vision_text_dual_encoder Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
visual_bert Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
vit ViT and Swin symbolic tracing with torch.fx (#17182) 2022-05-12 10:42:27 +02:00
vit_mae Fix ViTMAEModelTester (#17470) 2022-05-31 15:01:54 +02:00
wav2vec2 Black preview (#17217) 2022-05-12 16:25:55 -04:00
wav2vec2_conformer [Test] Fix W2V-Conformer integration test (#17303) 2022-05-17 18:20:36 +02:00
wav2vec2_phoneme Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
wav2vec2_with_lm Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
wavlm Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
xglm Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
xlm Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
xlm_prophetnet Black preview (#17217) 2022-05-12 16:25:55 -04:00
xlm_roberta Black preview (#17217) 2022-05-12 16:25:55 -04:00
xlm_roberta_xl Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
xlnet Fx support for multiple model architectures (#17393) 2022-05-31 10:02:55 +02:00
yolos Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
yoso Move test model folders (#17034) 2022-05-03 14:42:02 +02:00
__init__.py Move test model folders (#17034) 2022-05-03 14:42:02 +02:00