transformers/utils
eustlb 798f948e88
Add CSM model (#36719)
* draft structure

* depth decoder with forward pre hook

* full model forward draft

* draft update

* depth decoder update

* ConversationalSpeechModelForCausalLM udpates

* add generate

* max length criteria small fix

* udpate

* updates

* generation update

* update in loss compute

* conversion script

* update for correct input embeddings

* handle interleaved rope

* update

* update

* update

* support compile

* update training

* add doc

* update doc

* correct inits

* ConversationalSpeechModel -> Csm

* conf update

* name update

* tests CsmForCausalLMTest

* convert use cached_file

* conf + modeling updates

* generate utils handle third dim shape

* integration test

* modeling + conf updates

* common test handle more than 2 dims

* add nested audio list utils

* processing handle nested audio list

* csm processing draft

* mimi util

* init updates

* modular update

* convert modular

* processing update

* csm tests update

* generate tests handle third dim

* generate utils handle third dim

* propagate _get_initial_cache_position update

* tied_weight_keys update + convert correctly

* fix inputs_embeds

* revert audio nested list

* batch inference update + return audio

* audio_utils update

* processor update

* some more integration tests

* remove old test

* porcessing output labels

* improve

* fix

* update rope values with equivalent ones

* conversion update

* udpate tests

* handle depth decoder generation config

* remove default eos_token_id

* make style

* revert modeling_mimi

* add default generation_config

* remove sdpa since handled by default

* make

* fix conflict

* fix conflicts

* correct naming

* correct imports

* make

* causal -> conditional naming

* causal -> conditional naming

* auto update

* make

* make

* add doc

* test update

* fix weight init

* audio tokens offsets as buffer

* 4d mask in conditional class

* make

* doc update

* fix causal mask

* fix causal mask

* doc update

* doc update

* add processor doc

* update doc

* fix 4d causal mask

* update make_list_of_audio

* do not default to mutable

* remove duplicates

* remove useless reset_parameters

* use GradientCheckpointingLayer

* use can_return_tuple

* formatting

* prepend placeholder in _sample

* torch compile fix

* some more fixies

* convert modular

* fix

* default max_length in convert

* handle depth decoder generation config correctly

* clearer formulation

* handle output_loading_info

* handle softmax warning

* add doc

* propagate _get_initial_cache_position changes

* generation in its own module

* add processor tests

* fix compile witu cuda graphs

* fix compile with cuda graphs

* add csm.md

* include CSM loss

* doc nit

* doc nit

* doc nit

* Update docs/source/en/model_doc/csm.md

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* add save_audio to processor

* Update src/transformers/models/csm/modular_csm.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* doc update

* simplify audio_codes_mask computation

* doc update

* simplify loss computation

* fix static cache test

* fix

* remove comment

* simplify encoded length computation

* use hf-internal-testing

* doc update

* cast to float before numpy

* nit

* mem efficient codebook head

* nit

* cat input values with cutoffs

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2025-05-07 10:20:13 -04:00
..
test_module AutoImageProcessor (#20111) 2022-11-08 19:54:41 +00:00
tf_ops Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
add_pipeline_model_mapping_to_test.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
check_bad_commit.py Fix utils/check_bad_commit.py (#37272) 2025-04-04 12:18:20 +02:00
check_build.py Use deformable_detr kernel from the Hub (#36853) 2025-03-21 13:08:47 +01:00
check_config_attributes.py Add D-FINE Model into Transformers (#36261) 2025-04-29 12:17:55 +01:00
check_config_docstrings.py Add GraniteMoeHybrid support for 4.0 (#37658) 2025-05-06 06:47:43 +02:00
check_copies.py Fix typos (#37978) 2025-05-06 14:45:20 +01:00
check_doc_toc.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
check_docstrings.py Samhq model addition (#35147) 2025-04-28 19:07:09 +02:00
check_doctest_list.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
check_dummies.py Add llama4 (#37307) 2025-04-05 22:02:22 +02:00
check_inits.py Simplify soft dependencies and update the dummy-creation process (#36827) 2025-04-11 11:08:36 +02:00
check_model_tester.py Add a new script to check model testers' config (#22063) 2023-03-13 19:11:19 +01:00
check_modular_conversion.py Fix wrong argparse type in modular checker script (#37472) 2025-04-14 16:11:29 +01:00
check_repo.py Add CSM model (#36719) 2025-05-07 10:20:13 -04:00
check_self_hosted_runner.py Tiny fix for check_self_hosted_runner.py (#24052) 2023-06-06 18:17:41 +02:00
check_tf_ops.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
create_dependency_mapping.py Modular Conversion --fix_and_overwrite on Windows (#36583) 2025-03-06 13:12:30 +00:00
create_dummy_models.py CI: fix efficientnet pipeline timeout and prevent future similar issues due to large image size (#33123) 2024-08-27 11:58:27 +01:00
custom_init_isort.py chore: fix typos in utils module (#36668) 2025-03-13 15:12:44 +00:00
deprecate_models.py chore: fix typos in utils module (#36668) 2025-03-13 15:12:44 +00:00
download_glue_data.py Update ruff to 0.11.2 (#36962) 2025-03-25 16:00:11 +01:00
extract_warnings.py update github actions packages' version to suppress warnings (#30249) 2024-04-15 15:08:09 +02:00
fetch_hub_objects_for_ci.py Try to avoid/reduce some remaining CI job failures (#37202) 2025-04-02 14:39:57 +02:00
get_ci_error_statistics.py Add artifact name in job step to maintain job / artifact correspondence (#28682) 2024-01-31 15:58:17 +01:00
get_github_job_time.py Update ruff to 0.11.2 (#36962) 2025-03-25 16:00:11 +01:00
get_modified_files.py exclude deleted files in the fixup script (#21436) 2023-02-03 12:57:02 -05:00
get_previous_daily_ci.py Ping team members for new failed tests in daily CI (#34171) 2024-10-17 16:11:52 +02:00
get_test_info.py CI: fix efficientnet pipeline timeout and prevent future similar issues due to large image size (#33123) 2024-08-27 11:58:27 +01:00
important_models.txt ENH: [CI] Add new workflow to run slow tests of important models on push main if they are modified (#29235) 2024-04-12 10:01:28 +02:00
models_to_deprecate.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
modular_model_converter.py [modular] Fix the prefix-based renaming if the old and new model share a common name suffix (#37829) 2025-04-29 10:43:23 +02:00
not_doctested.txt Samhq model addition (#35147) 2025-04-28 19:07:09 +02:00
notification_service_doc_tests.py Refactor doctest (#30210) 2024-04-15 13:20:36 +02:00
notification_service_quantization.py Update ruff to 0.11.2 (#36962) 2025-03-25 16:00:11 +01:00
notification_service.py add job links to new model failure report (#37973) 2025-05-06 15:10:29 +02:00
past_ci_versions.py Update ruff to 0.11.2 (#36962) 2025-03-25 16:00:11 +01:00
patch_helper.py [Patch helper] update to not have to checkout main (#34006) 2024-10-09 09:21:46 +02:00
pr_slow_ci_models.py notify new model merged to main (#36375) 2025-02-24 17:53:18 +01:00
print_env.py Print more library versions in CI (#17384) 2022-06-02 10:24:16 +02:00
process_bad_commit_report.py add job links to new model failure report (#37973) 2025-05-06 15:10:29 +02:00
process_circleci_workflow_test_reports.py Update ruff to 0.11.2 (#36962) 2025-03-25 16:00:11 +01:00
process_test_artifacts.py fix the parallel number of CI nodes when it is smaller than number of tests (#33276) 2024-09-03 16:53:21 +02:00
release.py Remove research projects (#36645) 2025-03-11 13:47:38 +00:00
set_cuda_devices_for_ci.py Fix Cohere CI (#31263) 2024-06-10 15:16:58 +02:00
slow_documentation_tests.txt Update CodeLlama references (#30218) 2024-05-09 22:57:52 +02:00
sort_auto_mappings.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
split_doctest_jobs.py chore: fix typos in utils module (#36668) 2025-03-13 15:12:44 +00:00
split_model_tests.py consistent job / pytest report / artifact name correspondence (#30392) 2024-04-24 22:32:42 +02:00
tests_fetcher.py Add Optional to remaining types (#37808) 2025-04-28 14:20:45 +01:00
update_metadata.py Update ruff to 0.11.2 (#36962) 2025-03-25 16:00:11 +01:00
update_tiny_models.py Mention model_info.id instead of model_info.modelId (#32106) 2024-07-22 14:14:47 +01:00