Llama 3.1 8b Instruct - Memory Usage More than Reported

jacobvinje
Fri, 14 Feb 2025 14:39:28 +0000

@blakemart15 Which version of torch and transformers are you using?
I am only able to have a context window of 1K on my 8 A100 40GB GPUs, since it’s using LlamaAttention instead of LlamaSdpaAttention by default, which I think is because of something with my environment.