Llama 3.1 8b Instruct - Memory Usage More than Reported

blakemart15
Fri, 14 Feb 2025 13:42:12 +0000

@John6666 @jacobvinje Thank you for your replies

I originally did the tests on an A100 with nvidia driver version 535.183.06 and CUDA 12.2. This is where I was using all 80gb at ~60k tokens. I have since installed the cuda toolkit and am currently running nvidia drivers 570.86.10 and cuda 12.8. This has allowed me to run 128k tokens while only using ~50gb of VRAM.

As per the attention question, I was using the default attention. But with the cuda toolkit I am able to use fast attention 2. I have not seen any substantial memory or speed improvements from it.

Still very interesting that we are not close to that blog post