Finetuned LLM model conversion to GGUF - performance drop

meowman
Wed, 26 Jun 2024 22:01:18 +0000

That’s because you quantized the model.

Quantization reduces the precision of the model’s parameters from floating-point representations to lower-bit representations.

Lower bit numbers mean using fewer digits to represent a number. It’s like when you count from 1 to 10 instead of 1 to 100. So, lower bit numbers are simpler and take up less space!