Multi-gpu inference llama-3.2 vision with QLoRA

John6666
Wed, 16 Apr 2025 02:21:31 +0000

Well, since Accelerate is called with device_map=, it’s probably a problem with Accelerate and bitsandbytes…

And Llama 3.x sometimes has bugs when it’s not bfloat16. Also, device_map=“auto” doesn’t seem to get along well with bitsandbytes.

Well, in your case, it seems like you have enough VRAM even without device_map=, so I think that’s fine…

model = MllamaForConditionalGeneration.from_pretrained(
     model_path_or_name,
     torch_dtype=torch.bfloat16, # added
     quantization_config=bnb_config,
     #device_map='auto',
     device_map='sequential',
     low_cpu_mem_usage=True
 )

huggingface.co

Loading big models into memory

We’re on a journey to advance and democratize artificial intelligence through open source and open science.