transformers/docs/source/en
Younes Belkada 44b5506d29
[Llava] Add Llava to transformers (#27662)
* add model like

* logits match

* minor fixes

* fixes

* up

* up

* add todo

* llava processor

* keep the processor simple

* add conversion script

* fixup

* fix copies

* up

* add to index

* fix config + logits

* fix

* refactor

* more refactor

* more refactor

* fix copies

* add authors

* v1 tests

* add `LlavaProcessor` in init

* remove unneeded import

* up

* up

* docs

* up

* fix CI

* fix CI

* add attention  mask in test

* make fixup

* remove the vision model

* that' s the dirty way to do it

* nits

* nits

* updates

* add more tests

* add input tests

* fixup

* more styling

* nits

* updates amd cleanup

* fixup the generation expected results

* fix the testing script

* some cleanup and simplification which does not work yet but almost there!

* make correct dispatch operations

* vectorize works for batch of images and text

* last todos

* nits

* update test and modeling code

* remove useless function for now

* fix few issues

* fix generation

* some nits

* add bakllava

* nits

* remove duplicated code

* finis merge

* cleanup

* missed this line

* fill the todos

* add left padding offset

* add left and rignt padding logic

* bool to properly index

* make sure

* more cleanups

* batch is fixed 😉

* add correct device for tensor creation

* fix some dtype missmatch

* ruff

* update conversion script

* Update src/transformers/__init__.py

* fa 2 support + fix conversion script

* more

* correct reshaping

* fix test dict

* fix copies by ignoring

* fix nit

* skip clip vision model

* fixup

* fixup

* LlavaForVisionText2Text -> LlavaForCausalLM

* update

* fix

* raise correct errors

* fix

* docs

* nuke for now

* nits here and there

* fixup

* fix remaining tests

* update LlavaForConditionalGeneration instead of CausalLM

* fixups

* pipeline support

* slow and piepline tests

* supports batch

* nits

* cleanup

* fix first integration tests

* add pad token where needed

* correct etsts

* fixups

* update pipeline testr

* fix quality

* nits

* revert unneeded change

* nit

* use BatchFeature

* from ...feature_extraction_utils import BatchFeature

* nits

* nits

* properly update

* more f*** nits

* fix copies

* comment

* keep slow test slow

* Update src/transformers/models/llava/processing_llava.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* add piepline example

* add pixel values in docstrign

* update pr doctest

* fix

* fix slow tests

* remove hack

* fixup

* small note

* forward contrib credits from PR25789

* forward contrib credits from original implementation and work

* add arthur

* Update src/transformers/models/llava/processing_llava.py

Co-authored-by: Lysandre Debut <hi@lysand.re>

* update docstring

* nit

* move to not doctested because of timeout issues

* fixup

* add description

* more

* fix-copies

* fix docs

* add beam search

* add more comments

* add typehints on processor

* add speedup plot

* update slow tests and docs

* push test

* push batched test

* fix batched generation with different number of images

* remove benchmark due to a bug

* fix test

* fix copies

* add gcolab demo

---------

Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Co-authored-by: shauray8 <shauray8@users.noreply.github.com>
Co-authored-by: haotian-liu <haotian-liu@users.noreply.github.com>
Co-authored-by: Lysandre Debut <hi@lysand.re>
2023-12-07 09:30:47 +01:00
..
internal translate internal folder files to chinese (#27638) 2023-12-04 10:04:28 -08:00
main_classes [docs] Quantization (#27641) 2023-11-28 08:41:47 -08:00
model_doc [Llava] Add Llava to transformers (#27662) 2023-12-07 09:30:47 +01:00
tasks Translate en/tasks folder docs to Japanese 🇯🇵 (#27098) 2023-12-04 14:10:54 -08:00
_config.py [Styling] stylify using ruff (#27144) 2023-11-16 17:43:19 +01:00
_redirects.yml Extended semantic segmentation to image segmentation (#27039) 2023-11-23 15:58:21 +00:00
_toctree.yml [Llava] Add Llava to transformers (#27662) 2023-12-07 09:30:47 +01:00
accelerate.md Fix typos (#25936) 2023-09-04 11:15:12 +01:00
add_new_model.md Update add_new_model.md (#26365) 2023-09-25 12:58:11 +02:00
add_new_pipeline.md Update add_new_pipeline.md (#26197) 2023-09-19 00:41:16 +02:00
add_tensorflow_model.md Remove utils/documentation_tests.txt (#26213) 2023-09-18 13:33:01 +02:00
attention.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
autoclass_tutorial.md Update autoclass_tutorial.md (#25929) 2023-09-04 11:16:49 +01:00
benchmarks.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
bertology.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
big_models.md Fix typos (#25936) 2023-09-04 11:15:12 +01:00
chat_templating.md Update chat template warnings/guides (#27634) 2023-11-27 18:40:10 +00:00
community.md Update community.md (#25928) 2023-09-04 11:16:34 +01:00
contributing.md Enable doc in Spanish (#16518) 2022-04-04 10:25:46 -04:00
create_a_model.md [docs] fixed links with 404 (#27327) 2023-11-06 19:45:03 +00:00
custom_models.md Reorder the code on the Hub to explicit that sharing on the Hub isn't a requirement (#27691) 2023-11-27 09:38:18 +01:00
custom_tools.md [doc] Always call it Agents for consistency (#25958) 2023-09-05 12:27:20 +01:00
debugging.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
fast_tokenizers.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
generation_strategies.md [docs] navigation improvement between text gen pipelines and text gen params (#26477) 2023-09-29 09:43:39 +02:00
glossary.md [docs] Performance docs refactor p.2 (#26791) 2023-10-24 13:10:06 -04:00
hpo_train.md Remove-auth-token (#27060) 2023-11-13 14:20:54 +01:00
index.md [Llava] Add Llava to transformers (#27662) 2023-12-07 09:30:47 +01:00
installation.md [docs] Update offline mode docs (#26478) 2023-09-29 09:42:21 +02:00
llm_tutorial_optimization.md Generate: Update docs regarding reusing past_key_values in generate (#27612) 2023-11-21 10:48:14 +00:00
llm_tutorial.md Generate: update basic llm tutorial (#26937) 2023-10-19 16:53:28 +01:00
model_memory_anatomy.md Fix typos (#25936) 2023-09-04 11:15:12 +01:00
model_sharing.md Fix typos (#25936) 2023-09-04 11:15:12 +01:00
model_summary.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
multilingual.md Fix typo in example code (#25583) 2023-08-18 07:58:59 +02:00
notebooks.md Enable doc in Spanish (#16518) 2022-04-04 10:25:46 -04:00
pad_truncation.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
peft.md [Peft] modules_to_save support for peft integration (#27466) 2023-11-14 10:32:57 +01:00
perf_hardware.md docs: replace torch.distributed.run by torchrun (#27528) 2023-11-27 16:26:33 +00:00
perf_infer_cpu.md [docs] Update CPU/GPU inference docs (#26881) 2023-10-31 09:44:51 -07:00
perf_infer_gpu_one.md Flash Attention 2 support for RoCm (#27611) 2023-12-04 21:52:17 +09:00
perf_torch_compile.md Fix rendering for torch.compile() docs (#25432) 2023-08-10 13:25:00 +02:00
perf_train_cpu_many.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
perf_train_cpu.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
perf_train_gpu_many.md docs: replace torch.distributed.run by torchrun (#27528) 2023-11-27 16:26:33 +00:00
perf_train_gpu_one.md Reflect RoCm support in the documentation (#27636) 2023-11-25 00:59:17 +09:00
perf_train_special.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
perf_train_tpu_tf.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
perf_train_tpu.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
performance.md [docs] Update CPU/GPU inference docs (#26881) 2023-10-31 09:44:51 -07:00
perplexity.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
philosophy.md [docs] fixed links with 404 (#27327) 2023-11-06 19:45:03 +00:00
pipeline_tutorial.md [ASR Pipe] Improve docs and error messages (#26476) 2023-09-29 18:32:37 +01:00
pipeline_webserver.md Suggestions on Pipeline_webserver (#25570) 2023-08-18 10:17:44 +02:00
pr_checks.md Docstring check (#26052) 2023-10-04 15:13:37 +02:00
preprocessing.md Broken links fixed related to datasets docs (#27569) 2023-11-17 13:44:09 -08:00
quantization.md [Docs] Update broken image on fused modules (#27856) 2023-12-05 12:33:58 -08:00
quicktour.md [TYPO] fix typo/format in quicktour.md (#25519) 2023-08-16 08:03:23 +02:00
run_scripts.md docs: replace torch.distributed.run by torchrun (#27528) 2023-11-27 16:26:33 +00:00
sagemaker.md [docs] fixed links with 404 (#27327) 2023-11-06 19:45:03 +00:00
serialization.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
task_summary.md Fix doctest (#25031) 2023-07-25 22:10:06 +02:00
tasks_explained.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
testing.md Device agnostic testing (#25870) 2023-10-24 16:49:26 +02:00
tf_xla.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
tflite.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
tokenizer_summary.md Fix typo: Roberta -> RoBERTa (#25302) 2023-08-03 14:17:30 -07:00
torchscript.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00
training.md Fix semantic error in evaluation section (#27675) 2023-11-24 12:41:16 +01:00
transformers_agents.md [doc] Always call it Agents for consistency (#25958) 2023-09-05 12:27:20 +01:00
troubleshooting.md Migrate doc files to Markdown. (#24376) 2023-06-20 18:07:47 -04:00