transformers/tests
Pi Esposito fb2b45e562
add in layer gpt2 tokenizer (#20421)
* add minimal working gpt2 tokenizer

* graph mode and output equivalence tests working

* not today tensorflow. serialization test passing!

* fix style, documentation, docstrings and all that jazz

* passing consistency checks

* move keras nlp to tf dependencies

* fix tf modeling utils and gpt2 attention to enable compiling

* fix (I hope) keras nlp dependencies

* rever changes on generation

* remove debug prints

* remove redundant tf dummy objects

* add from config, get config and max length settings to address review

* let flake ignore the error on distillation you are welcome

* test from config

* add padding test

* address sgugger review
2022-11-29 10:02:40 -05:00
..
benchmark [Test refactor 1/5] Per-folder tests reorganization (#15725) 2022-02-23 15:46:28 -05:00
deepspeed Fix tapas scatter (#20149) 2022-11-14 01:04:26 -05:00
extended Update self-push workflow (#17177) 2022-05-13 16:28:00 +02:00
fixtures add a warning in SpmConverter for sentencepiece's model using the byte fallback feature (#16629) 2022-04-11 11:06:10 +02:00
generation Generate: model_kwargs can also be an input to prepare_inputs_for_generation (#20353) 2022-11-21 16:20:27 +00:00
mixed_int8 [BNB] Throw ValueError when trying to cast or assign (#20409) 2022-11-23 15:51:50 +01:00
models add in layer gpt2 tokenizer (#20421) 2022-11-29 10:02:40 -05:00
onnx Use tiny models for ONNX tests - text modality (#20333) 2022-11-22 17:11:17 +01:00
optimization [Test refactor 1/5] Per-folder tests reorganization (#15725) 2022-02-23 15:46:28 -05:00
pipelines Add Audio Spectogram Transformer (#19981) 2022-11-21 18:58:54 +01:00
repo_utils Repo utils test (#19696) 2022-10-18 13:47:36 -04:00
sagemaker transformers-cli login => huggingface-cli login (#18490) 2022-08-06 09:42:55 +02:00
tokenization fix train_new_from_iterator in the case of byte-level tokenizers (#17549) 2022-06-08 15:30:41 +02:00
trainer [AnyPrecisionAdamW] test fix (#20454) 2022-11-25 09:02:10 -08:00
utils Include image processor in add-new-model-like (#20439) 2022-11-28 16:46:02 +00:00
__init__.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_configuration_common.py Add WhisperModel to transformers (#19166) 2022-10-05 22:28:31 +02:00
test_feature_extraction_common.py Add tests for legacy load by url and fix bugs (#19078) 2022-09-16 23:20:02 +02:00
test_image_transforms.py [Image Transformers] to_pil fix float edge cases (#20406) 2022-11-23 13:47:59 +01:00
test_modeling_common.py [AutoBackbone] Improve API (#20407) 2022-11-28 17:20:24 +01:00
test_modeling_flax_common.py Allow flax subfolder (#19902) 2022-10-26 18:33:23 +02:00
test_modeling_tf_common.py More TF int dtype fixes (#20384) 2022-11-28 13:24:44 +00:00
test_sequence_feature_extraction_common.py Some tests misusing assertTrue for comparisons fix (#16771) 2022-04-19 14:44:08 +02:00
test_tokenization_common.py 🚨 🚨 🚨 Fix Issue 15003: SentencePiece Tokenizers Not Adding Special Tokens in convert_tokens_to_string (#15775) 2022-11-02 15:45:38 -04:00