mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-31 02:02:21 +06:00
a2eb75c891
* Support `flash_attn_3` Implements fwd and tests for Flash Attention 3 https://github.com/Dao-AILab/flash-attention/commits/main/hopper - Includes checks for dropout>0 and ALiBi in `modeling_utils.PreTrainedModel._check_and_enable_flash_attn_3` (Dropout will likely be supported soon, so this will need to be updated and `modeling_flash_attention_utils._flash_attention_forward` at the `if _IS_FLASH_ATTN_3_AVAILABLE: ...` An example Llama implementation is included in `modeling_llama.py` but other models would still need to be updated Based on https://github.com/huggingface/transformers/pull/36190 which has model implementations and examples which could be merged * Add tests for Flash Attention 2 and 3 parity * ci fix * FA2 compatibiity - `_prepare_flash_attention_from_position_ids` ->`prepare_fa2_from_position_ids` - Remove bettertransformer check in Flash Attention 3 - Merge tests - Add licensing * ci fix * Test naming consistency * ci fix * Deprecation warning for `prepare_fa2_from_position_ids` * ci fix |
||
|---|---|---|
| .. | ||
| import_structures | ||
| __init__.py | ||
| test_activations_tf.py | ||
| test_activations.py | ||
| test_add_new_model_like.py | ||
| test_audio_utils.py | ||
| test_auto_docstring.py | ||
| test_backbone_utils.py | ||
| test_cache_utils.py | ||
| test_chat_template_utils.py | ||
| test_cli.py | ||
| test_configuration_utils.py | ||
| test_convert_slow_tokenizer.py | ||
| test_deprecation.py | ||
| test_doc_samples.py | ||
| test_dynamic_module_utils.py | ||
| test_expectations.py | ||
| test_feature_extraction_utils.py | ||
| test_file_utils.py | ||
| test_generic.py | ||
| test_hf_argparser.py | ||
| test_hub_utils.py | ||
| test_image_processing_utils.py | ||
| test_image_utils.py | ||
| test_import_structure.py | ||
| test_import_utils.py | ||
| test_logging.py | ||
| test_model_card.py | ||
| test_model_debugging_utils.py | ||
| test_model_output.py | ||
| test_modeling_rope_utils.py | ||
| test_modeling_utils.py | ||
| test_offline.py | ||
| test_processing_utils.py | ||
| test_skip_decorators.py | ||
| test_tokenization_utils.py | ||
| test_versions_utils.py | ||
| test_video_utils.py | ||
| tiny_model_summary.json | ||