riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-02 03:01:07 +06:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

Make clearer about zero_init requirements (#29879)

Browse Source 
* Docstring to note about zero init

* Check for accelerate

* Change conditional return

* Tweak

* Add new accelerate-specific zero3 check

* Fix import

* Revert to RTFM

* Update src/transformers/modeling_utils.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
This commit is contained in:
Zach Mueller 2024-04-03 13:37:52 -04:00 committed by GitHub
parent 695d823323
commit 863e2562d8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
src/transformers/training_args.py
View File

@ -504,6 +504,11 @@ class TrainingArguments:
evolve in the future. The value is either the location of DeepSpeed json config file (e.g., evolve in the future. The value is either the location of DeepSpeed json config file (e.g.,
`ds_config.json`) or an already loaded json file as a `dict`" `ds_config.json`) or an already loaded json file as a `dict`"
<Tip warning={true}>
If enabling any Zero-init, make sure that your model is not initialized until
*after* initializing the `TrainingArguments`, else it will not be applied.
</Tip>
accelerator_config (`str`, `dict`, or `AcceleratorConfig`, *optional*): accelerator_config (`str`, `dict`, or `AcceleratorConfig`, *optional*):
Config to be used with the internal `Accelerator` implementation. The value is either a location of Config to be used with the internal `Accelerator` implementation. The value is either a location of
accelerator json config file (e.g., `accelerator_config.json`), an already loaded json file as `dict`, accelerator json config file (e.g., `accelerator_config.json`), an already loaded json file as `dict`,