Llama 3.1 8b Instruct - Memory Usage More than Reported

John6666
Fri, 14 Feb 2025 01:10:18 +0000

I don’t think there’s anything wrong with the code. The blog content is sometimes inaccurate, but 8B shouldn’t cause a VRAM shortage…
Why not try a different 8B-class LLM to see if it’s Llama causing the problem?

github.com/meta-llama/llama

7B model CUDA out of memory on rtx3090ti 24Gb

opened 08:09AM - 06 Mar 23 UTC

closed 06:59PM - 15 Aug 23 UTC

Jehuty-ML

i have seen someone in this issues Message area said that 7B model just needs 8.…5G VRAM. but why i ran the example.py returns out of memory on a 24G VRAM cards? any help will be appreciated! Thanks!