Llama 3.1 8b Instruct - Memory Usage More than Reported

jacobvinje
Fri, 14 Feb 2025 12:54:31 +0000

I also read the same blog post and chose hardware for Llama 3.1 8B Instruct based on the numbers in the post. I have not been able to replicate the numbers from the post. However, when I have been able to use LlamaSdpaAttention, I have gotten close, but with LlamaAttention I am nowhere near close to the numbers in the blog post.
Which attention you’re using can be seen when you print(model). Would be curious to see which attention class your setup is using.