riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-01 02:31:11 +06:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

[docs] Improve visualization for vertical parallelism (#28583)

Browse Source 
The documentation says "We refer to this Model parallelism as “Vertical” because of how models are typically visualized.", but then visualizes the model horizontally. This change visualizes the model indeed vertically.
This commit is contained in:
Peter Götz 2024-01-25 18:55:11 +01:00 committed by GitHub
parent 4cbd876e42
commit 2875195887
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 13 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
docs/source/en/perf_train_gpu_many.md
View File

@ -285,10 +285,19 @@ following diagram shows an 8-layer model split vertically into two slices, placi
GPU0 and 4-7 to GPU1:
```
=================== ===================
| 0 | 1 | 2 | 3 | | 4 | 5 | 6 | 7 |
=================== ===================
GPU0 GPU1
================
| Layer | |
| 0 | |
| 1 | GPU0 |
| 2 | |
| 3 | |
================
| Layer | |
| 4 | |
| 5 | GPU1 |
| 6 | |
| 7 | |
================
```
In this example, when data moves from layer 0 to 3, it's no different from regular forward pass. However, passing data