riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity

Fix DBRX LayerNorm init method (#35177)

Browse Source 
fix dbrx layernorm init
This commit is contained in:
Huang, Guangtai 2024-12-10 06:31:22 -08:00 committed by GitHub
parent 5fba3f99c0
commit 3e2769a3c9
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/models/dbrx/modeling_dbrx.py
View File

@ -845,7 +845,7 @@ class DbrxPreTrainedModel(PreTrainedModel):
if module.padding_idx is not None:
module.weight.data[module.padding_idx].zero_()
elif isinstance(module, nn.LayerNorm):
module.weight.data.normal_(mean=0.0, std=std)
module.weight.data.fill_(1.0)
if module.bias is not None:
module.bias.data.zero_()
elif isinstance(module, DbrxExpertGLU):