transformers/docs/source/en
Raushan Turganbay 187439c3fa
VLM: special multimodal Tokenizer (#34461)
* kinda works

* update

* add tests

* update

* use special tokens in processors

* typo

* fix copies

* fix

* fix moshi after rebase

* update

* fix tests

* update

* Update docs/source/en/main_classes/tokenizer.md

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* update docs

* test for load time adding tokens

* fix some more tests which are now fetched better

* one more fix

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2024-11-04 16:37:51 +01:00
..
internal SynthID: better example (#34372) 2024-10-25 11:46:46 +01:00
main_classes VLM: special multimodal Tokenizer (#34461) 2024-11-04 16:37:51 +01:00
model_doc Add Image Processor Fast RT-DETR (#34354) 2024-10-30 13:49:47 -04:00
quantization [Docs] Update compressed_tensors.md (#33961) 2024-10-10 15:22:41 +02:00
tasks Add post_process_depth_estimation to image processors and support ZoeDepth's inference intricacies (#32550) 2024-10-22 15:50:54 +02:00
_config.py [Doc]: Broken link in Kubernetes doc (#33879) 2024-10-04 11:20:56 +02:00
_redirects.yml
_toctree.yml Zamba is an LM (#34342) 2024-10-24 14:29:33 +02:00
accelerate.md
add_new_model.md Model addition timeline (#33762) 2024-09-27 17:15:13 +02:00
add_new_pipeline.md
agents_advanced.md Fix typos in agents_advanced.md (#34405) 2024-10-25 08:52:29 -07:00
agents.md Fix method name which changes in tutorial (#34252) 2024-10-21 14:21:52 -03:00
attention.md
autoclass_tutorial.md
benchmarks.md
bertology.md
big_models.md
chat_templating.md Add a doc section on writing generation prompts (#34248) 2024-10-21 14:35:57 +01:00
community.md
contributing.md
conversations.md
create_a_model.md
custom_models.md
debugging.md
deepspeed.md
fast_tokenizers.md
fsdp.md
generation_strategies.md Universal Assisted Generation: Assisted generation with any assistant model (by Intel Labs) (#33383) 2024-10-10 14:41:53 +02:00
gguf.md Add GGUF for Mamba (#34200) 2024-10-30 16:52:17 +01:00
glossary.md
how_to_hack_models.md [Docs] Add Developer Guide: How to Hack Any Transformers Model (#33979) 2024-10-07 10:08:20 +02:00
hpo_train.md Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00
index.md add code generation to natural language processing section (#34333) 2024-10-24 14:42:47 +02:00
installation.md
kv_cache.md
llm_optims.md Remove graph breaks for torch.compile() in flash_attention_forward when Lllama Model is padding free tuned (#33932) 2024-10-24 11:02:54 +02:00
llm_tutorial_optimization.md
llm_tutorial.md Fix: typo (#33880) 2024-10-02 09:12:21 +01:00
model_memory_anatomy.md
model_sharing.md
model_summary.md
modular_transformers.md Improve modular converter (#33991) 2024-10-08 14:53:58 +02:00
multilingual.md
notebooks.md
pad_truncation.md
peft.md
perf_hardware.md
perf_infer_cpu.md
perf_infer_gpu_one.md Attn implementation for composite models (#32238) 2024-10-22 06:54:44 +02:00
perf_torch_compile.md
perf_train_cpu_many.md update doc (#34478) 2024-10-31 15:59:23 -07:00
perf_train_cpu.md update doc (#34478) 2024-10-31 15:59:23 -07:00
perf_train_gpu_many.md
perf_train_gpu_one.md
perf_train_special.md
perf_train_tpu_tf.md
performance.md
perplexity.md Fix perplexity computation in perplexity.md (#34387) 2024-10-29 11:10:10 +01:00
philosophy.md
pipeline_tutorial.md
pipeline_webserver.md
pr_checks.md
preprocessing.md
quicktour.md [docs] fix typo (#34235) 2024-10-22 09:46:07 -07:00
run_scripts.md
sagemaker.md
serialization.md
task_summary.md
tasks_explained.md
testing.md
tf_xla.md
tflite.md
tiktoken.md
tokenizer_summary.md
torchscript.md
trainer.md Adding optimizer_cls_and_kwargs to Trainer.__init__ (#34358) 2024-10-29 16:23:16 +01:00
training.md
troubleshooting.md