Multi-gpu inference llama-3.2 vision with QLoRA

shon-ot
Wed, 16 Apr 2025 07:41:50 +0000

Hi @John6666 thanks for your inputs.

I don’t think the issue lies in loading the model itself, as I can clearly see the model is being properly sharded across devices when using device_map='auto'. I should also mention that everything works perfectly when running inference with just the base model — it gets correctly sharded and the batch is distributed across the GPUs. The problem only arises after adding the PEFT adapter.