Stas Bekman 580dd87c55 [Deepspeed] add support for bf16 mode (#14569) ... * [WIP] add support for bf16 mode * prep for bf16 * prep for bf16 * fix; zero2/bf16 is ok * check bf16 is available * test fixes * enable zero3_bf16 * config files * docs * split stage_dtype; merge back to non-dtype-specific config file * fix doc * cleanup * cleanup * bfloat16 => bf16 to match the PR changes * s/zero_gather_fp16_weights_on_model_save/zero_gather_16bit_weights_on_model_save/; s/save_fp16_model/save_16bit_model/ * test fixes/skipping * move * fix * Update docs/source/main_classes/deepspeed.mdx Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * backticks * cleanup * cleanup * cleanup * new version * add note about grad accum in bf16 Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>		2022-03-11 17:53:53 -08:00
..
ds_config_zero2.json	[Deepspeed] add support for bf16 mode (#14569)	2022-03-11 17:53:53 -08:00
ds_config_zero3.json	[Deepspeed] add support for bf16 mode (#14569)	2022-03-11 17:53:53 -08:00
test_deepspeed.py	[Deepspeed] add support for bf16 mode (#14569)	2022-03-11 17:53:53 -08:00
test_model_zoo.py	[Deepspeed] add support for bf16 mode (#14569)	2022-03-11 17:53:53 -08:00