![]() * enable cpu bnb path
* fix style
* fix code style
* fix 4 bit path
* Update src/transformers/utils/import_utils.py
Co-authored-by: Aarni Koskela <akx@iki.fi>
* add multi backend refactor tests
* fix style
* tweak 4bit quantizer + fix corresponding tests
* tweak 8bit quantizer + *try* fixing corresponding tests
* fix dequant bnb 8bit
* account for Intel CPU in variability of expected outputs
* enable cpu and xpu device map
* further tweaks to account for Intel CPU
* fix autocast to work with both cpu + cuda
* fix comments
* fix comments
* switch to testing_utils.torch_device
* allow for xpu in multi-gpu tests
* fix tests 4bit for CPU NF4
* fix bug with is_torch_xpu_available needing to be called as func
* avoid issue where test reports attr err due to other failure
* fix formatting
* fix typo from resolving of merge conflict
* polish based on last PR review
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* fix CI
* Update src/transformers/integrations/integration_utils.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update src/transformers/integrations/integration_utils.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* fix error log
* fix error msg
* add \n in error log
* make quality
* rm bnb cuda restriction in doc
* cpu model don't need dispatch
* fix doc
* fix style
* check cuda avaliable in testing
* fix tests
* Update docs/source/en/model_doc/chameleon.md
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Update docs/source/en/model_doc/llava_next.md
Co-authored-by: Aarni Koskela <akx@iki.fi>
* Update tests/quantization/bnb/test_4bit.py
Co-authored-by: Aarni Koskela <akx@iki.fi>
* Update tests/quantization/bnb/test_4bit.py
Co-authored-by: Aarni Koskela <akx@iki.fi>
* fix doc
* fix check multibackends
* fix import sort
* remove check torch in bnb
* docs: update bitsandbytes references with multi-backend info
* docs: fix small mistakes in bnb paragraph
* run formatting
* reveret bnb check
* move bnb multi-backend check to import_utils
* Update src/transformers/utils/import_utils.py
Co-authored-by: Aarni Koskela <akx@iki.fi>
* fix bnb check
* minor fix for bnb
* check lib first
* fix code style
* Revert "run formatting"
This reverts commit
|
||
---|---|---|
.. | ||
__init__.py | ||
README.md | ||
test_4bit.py | ||
test_mixed_int8.py |
Testing mixed int8 quantization
The following is the recipe on how to effectively debug bitsandbytes
integration on Hugging Face transformers
.
Library requirements
transformers>=4.22.0
accelerate>=0.12.0
bitsandbytes>=0.31.5
.
Hardware requirements
The following instructions are tested with 2 NVIDIA-Tesla T4 GPUs. To run successfully bitsandbytes
you would need a 8-bit core tensor supported GPU. Note that Turing, Ampere or newer architectures - e.g. T4, RTX20s RTX30s, A40-A100, A6000 should be supported.
Virutal envs
conda create --name int8-testing python==3.8
pip install bitsandbytes>=0.31.5
pip install accelerate>=0.12.0
pip install transformers>=4.23.0
if transformers>=4.23.0
is not released yet, then use:
pip install git+https://github.com/huggingface/transformers.git
Troubleshooting
A list of common errors:
Torch does not correctly do the operations on GPU
First check that:
import torch
vec = torch.randn(1, 2, 3).to(0)
Works without any error. If not, install torch using conda
like:
conda create --name int8-testing python==3.8
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
pip install bitsandbytes>=0.31.5
pip install accelerate>=0.12.0
pip install transformers>=4.23.0
For the latest pytorch instructions please see this
and the snippet above should work.
bitsandbytes operations are not supported under CPU!
This happens when some Linear weights are set to the CPU when using accelerate
. Please check carefully model.hf_device_map
and make sure that there is no Linear
module that is assigned to CPU. It is fine to have the last module (usually the Lm_head) set on CPU.
To use the type as a Parameter, please correct the detach() semantics defined by __torch_dispatch__() implementation.
Use the latest version of accelerate
with a command such as: pip install -U accelerate
and the problem should be solved.
Parameter has no attribue .CB
Same solution as above.
RuntimeError: CUDA error: an illegal memory access was encountered ... consider passing CUDA_LAUNCH_BLOCKING=1
Run your script by pre-pending CUDA_LAUNCH_BLOCKING=1
and you should observe an error as described in the next section.
CUDA illegal memory error: an illegal memory access at line...
:
Check the CUDA verisons with:
nvcc --version
and confirm it is the same version as the one detected by bitsandbytes
. If not, run:
ls -l $CONDA_PREFIX/lib/libcudart.so
or
ls -l $LD_LIBRARY_PATH
Check if libcudart.so
has a correct symlink that is set. Sometimes nvcc
detects the correct CUDA version but bitsandbytes
doesn't. You have to make sure that the symlink that is set for the file libcudart.so
is redirected to the correct CUDA file.
Here is an example of a badly configured CUDA installation:
nvcc --version
gives:
which means that the detected CUDA version is 11.3 but bitsandbytes
outputs:
First check:
echo $LD_LIBRARY_PATH
If this contains multiple paths separated by :
. Then you have to make sure that the correct CUDA version is set. By doing:
ls -l $path/libcudart.so
On each path ($path
) separated by :
.
If not, simply run
ls -l $LD_LIBRARY_PATH/libcudart.so
and you can see
If you see that the file is linked to the wrong CUDA version (here 10.2), find the correct location for libcudart.so
(find --name libcudart.so
) and replace the environment variable LD_LIBRARY_PATH
with the one containing the correct libcudart.so
file.