Unable to load a FineTuned LLama Model to GPU for inference

abbbinav
Fri, 15 Dec 2023 08:54:52 +0000

Use Kaggle for testing the finetuned model as it provides 30GB of RAM and 2 T4 GPUs. also, try using !pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 and !pip install transformers==4.34.0, it helped me load the model partially onto the GPU.