transformers/utils
Younes Belkada 44b5506d29
[Llava] Add Llava to transformers (#27662)
* add model like

* logits match

* minor fixes

* fixes

* up

* up

* add todo

* llava processor

* keep the processor simple

* add conversion script

* fixup

* fix copies

* up

* add to index

* fix config + logits

* fix

* refactor

* more refactor

* more refactor

* fix copies

* add authors

* v1 tests

* add `LlavaProcessor` in init

* remove unneeded import

* up

* up

* docs

* up

* fix CI

* fix CI

* add attention  mask in test

* make fixup

* remove the vision model

* that' s the dirty way to do it

* nits

* nits

* updates

* add more tests

* add input tests

* fixup

* more styling

* nits

* updates amd cleanup

* fixup the generation expected results

* fix the testing script

* some cleanup and simplification which does not work yet but almost there!

* make correct dispatch operations

* vectorize works for batch of images and text

* last todos

* nits

* update test and modeling code

* remove useless function for now

* fix few issues

* fix generation

* some nits

* add bakllava

* nits

* remove duplicated code

* finis merge

* cleanup

* missed this line

* fill the todos

* add left padding offset

* add left and rignt padding logic

* bool to properly index

* make sure

* more cleanups

* batch is fixed 😉

* add correct device for tensor creation

* fix some dtype missmatch

* ruff

* update conversion script

* Update src/transformers/__init__.py

* fa 2 support + fix conversion script

* more

* correct reshaping

* fix test dict

* fix copies by ignoring

* fix nit

* skip clip vision model

* fixup

* fixup

* LlavaForVisionText2Text -> LlavaForCausalLM

* update

* fix

* raise correct errors

* fix

* docs

* nuke for now

* nits here and there

* fixup

* fix remaining tests

* update LlavaForConditionalGeneration instead of CausalLM

* fixups

* pipeline support

* slow and piepline tests

* supports batch

* nits

* cleanup

* fix first integration tests

* add pad token where needed

* correct etsts

* fixups

* update pipeline testr

* fix quality

* nits

* revert unneeded change

* nit

* use BatchFeature

* from ...feature_extraction_utils import BatchFeature

* nits

* nits

* properly update

* more f*** nits

* fix copies

* comment

* keep slow test slow

* Update src/transformers/models/llava/processing_llava.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* add piepline example

* add pixel values in docstrign

* update pr doctest

* fix

* fix slow tests

* remove hack

* fixup

* small note

* forward contrib credits from PR25789

* forward contrib credits from original implementation and work

* add arthur

* Update src/transformers/models/llava/processing_llava.py

Co-authored-by: Lysandre Debut <hi@lysand.re>

* update docstring

* nit

* move to not doctested because of timeout issues

* fixup

* add description

* more

* fix-copies

* fix docs

* add beam search

* add more comments

* add typehints on processor

* add speedup plot

* update slow tests and docs

* push test

* push batched test

* fix batched generation with different number of images

* remove benchmark due to a bug

* fix test

* fix copies

* add gcolab demo

---------

Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Co-authored-by: shauray8 <shauray8@users.noreply.github.com>
Co-authored-by: haotian-liu <haotian-liu@users.noreply.github.com>
Co-authored-by: Lysandre Debut <hi@lysand.re>
2023-12-07 09:30:47 +01:00
..
test_module AutoImageProcessor (#20111) 2022-11-08 19:54:41 +00:00
tf_ops Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
add_pipeline_model_mapping_to_test.py A script to add/update pipeline_model_mapping systematically (#22180) 2023-04-06 18:08:14 +02:00
check_build.py Clean up CUDA kernels (#23455) 2023-05-18 14:14:43 -04:00
check_config_attributes.py Add SeamlessM4T v2 (#27779) 2023-11-30 20:24:43 +01:00
check_config_docstrings.py [Check] Fix config docstring (#26222) 2023-09-18 19:58:01 +02:00
check_copies.py [Llava] Add Llava to transformers (#27662) 2023-12-07 09:30:47 +01:00
check_doc_toc.py Doc checks (#25408) 2023-08-10 10:53:22 +02:00
check_docstrings.py Add Llama Flax Implementation (#24587) 2023-12-07 07:05:00 +01:00
check_doctest_list.py Avoid many failing tests in doctesting (#27262) 2023-11-03 12:47:07 +01:00
check_dummies.py Doc checks (#25408) 2023-08-10 10:53:22 +02:00
check_inits.py Make using safetensors files automated. (#27571) 2023-12-01 15:51:10 +01:00
check_model_tester.py Add a new script to check model testers' config (#22063) 2023-03-13 19:11:19 +01:00
check_repo.py [Time series] Add PatchTSMixer (#26247) 2023-12-05 15:31:35 +01:00
check_self_hosted_runner.py Tiny fix for check_self_hosted_runner.py (#24052) 2023-06-06 18:17:41 +02:00
check_table.py [Llava] Add Llava to transformers (#27662) 2023-12-07 09:30:47 +01:00
check_task_guides.py More utils doc (#25457) 2023-08-17 07:58:35 +02:00
check_tf_ops.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
create_dummy_models.py Update tiny model creation script (#27674) 2023-11-28 10:05:34 +01:00
custom_init_isort.py More utils doc (#25457) 2023-08-17 07:58:35 +02:00
download_glue_data.py Raise exceptions instead of asserts (#13907) 2021-10-07 12:44:23 +05:30
extract_warnings.py Make Slack CI reporting stronger (#21823) 2023-02-28 17:12:44 +01:00
get_ci_error_statistics.py Show diff between 2 CI runs on Slack reports (#22798) 2023-04-19 19:27:37 +02:00
get_github_job_time.py Make Slack CI reporting stronger (#21823) 2023-02-28 17:12:44 +01:00
get_modified_files.py exclude deleted files in the fixup script (#21436) 2023-02-03 12:57:02 -05:00
get_previous_daily_ci.py Fix a minor bug in CI slack report (#22906) 2023-04-21 20:36:35 +02:00
get_test_info.py Add an utility file to get information from test files (#21856) 2023-03-01 17:53:29 +01:00
not_doctested.txt [Llava] Add Llava to transformers (#27662) 2023-12-07 09:30:47 +01:00
notification_service_doc_tests.py Fix slack report failing for doctest (#27042) 2023-10-30 10:48:24 +01:00
notification_service.py restructure AMD scheduled CI (#27743) 2023-12-04 15:32:05 +01:00
past_ci_versions.py (Re-)Enable Nightly + Past CI (#22393) 2023-03-30 21:06:35 +02:00
print_env.py Print more library versions in CI (#17384) 2022-06-02 10:24:16 +02:00
release.py More utils doc (#25457) 2023-08-17 07:58:35 +02:00
slow_documentation_tests.txt Add SeamlessM4T v2 (#27779) 2023-11-30 20:24:43 +01:00
sort_auto_mappings.py More utils doc (#25457) 2023-08-17 07:58:35 +02:00
tests_fetcher.py Trigger corresponding pipeline tests if tests/utils/tiny_model_summary.json is modified (#27693) 2023-11-28 17:21:21 +01:00
update_metadata.py Update processor mapping for hub snippets (#27477) 2023-11-14 20:05:54 +00:00
update_tiny_models.py Update tiny model summary file for recent models (#22637) 2023-04-06 22:52:59 +02:00