Finetuned LLM model conversion to GGUF - performance drop

tprochenka
Wed, 31 Jul 2024 06:13:18 +0000

Hi @meowman, if I do a conversion without quantization model is huge and inference times are endless…