Finetuned LLM model conversion to GGUF - performance drop

iamz80
Fri, 21 Jun 2024 17:49:53 +0000

Hello!

I have two concerns regarding your setup.

1. Are you sure alpha=16 isn’t too much?

2. When you merge, do you merge with the original non-quantized model or with a quantized version? My concern is that if LoRA is trained with the quantized version but merged with the original model, the model output might not be the same. And when it is quantized to GGUF format it adds on top of that additional error. And the two stacked together might be enough to degrade the output considerably.

Thank you!