Sourab Mangrulkar
|
bdfefbadaf
|
Update ds_config_zero3.json (#30829)
|
2024-05-15 10:02:31 -04:00 |
|
Stas Bekman
|
580dd87c55
|
[Deepspeed] add support for bf16 mode (#14569)
* [WIP] add support for bf16 mode
* prep for bf16
* prep for bf16
* fix; zero2/bf16 is ok
* check bf16 is available
* test fixes
* enable zero3_bf16
* config files
* docs
* split stage_dtype; merge back to non-dtype-specific config file
* fix doc
* cleanup
* cleanup
* bfloat16 => bf16 to match the PR changes
* s/zero_gather_fp16_weights_on_model_save/zero_gather_16bit_weights_on_model_save/; s/save_fp16_model/save_16bit_model/
* test fixes/skipping
* move
* fix
* Update docs/source/main_classes/deepspeed.mdx
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* backticks
* cleanup
* cleanup
* cleanup
* new version
* add note about grad accum in bf16
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
|
2022-03-11 17:53:53 -08:00 |
|
Stas Bekman
|
32290d87f6
|
[Deepspeed] various fixes (#12058)
* replace deprecated config
* sub_group_size was too big
* complete deprecation removal
|
2021-06-08 08:36:15 -07:00 |
|
Stas Bekman
|
bc2571e61c
|
[Deepspeed] ZeRO-Infinity integration plus config revamp (#11418)
* adding Z-inf
* revamp config process
* up version requirement
* wip
* massive rewrite
* cleanup
* cleanup
* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* consistent json commas
* act on suggestions
* leave this feature for 0.3.16
* style
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
|
2021-04-26 10:40:32 -07:00 |
|
Stas Bekman
|
66446909b2
|
[tests] relocate core integration tests (#11146)
* relocate core integration tests
* add sys.path context manager
* cleanup
* try
* try2
* fix path
* doc
* style
* add dep
* add 2 more deps
|
2021-04-08 13:13:17 -07:00 |
|