Fine Tuning LLama 3.2 1B Quantized Memory Requirements

BenjaminB
Fri, 29 Nov 2024 15:14:08 +0000

With the given information, it’s hard to tell what the reason is for why so much memory is needed, there is nothing obviously wrong there. It would be helpful if you could share the full code and the full error message.

One common source of excessive memory usage can be in the data itself. If you have very long sequences, because of the quadratic memory requirement, OOMs can easily occur. You could check if setting a low max_seq_length helps to curb the memory usage.