Finetuned LLM model conversion to GGUF - performance drop

tprochenka
Wed, 31 Jul 2024 06:11:37 +0000

HI @iamz80, thank you for your insights and sorry for the late response.
ad 1. I tested different configurations of alpha and r - and it is ok. When is a best setup in your opinion? I saw different opinions.
ad 2. When I fine tune a model a load a base model with quantization:

            model = AutoPeftModelForCausalLM.from_pretrained(
                model_path,
                quantization_config=bnb_config,
                device_map={"": 0},
                trust_remote_code=True,
                use_cache=True,
                is_trainable=False,
            )

Then adapter is bf16 or tf32:

    args = TrainingArguments(
        output_dir=OUTPUT_MODEL_DIR,
        num_train_epochs=ft_parameters.EPOCHS,
        per_device_train_batch_size=ft_parameters.BATCH_SIZE,  # 4 --> 2 becouse gpu
        per_device_eval_batch_size=1,
        gradient_accumulation_steps=ft_parameters.ACCUMULATION_STEP,
        gradient_checkpointing=True,
        optim="paged_adamw_32bit",
        logging_steps=10,
        save_strategy="epoch",
        learning_rate=ft_parameters.LEARINIG_RATE,
        bf16=True,
        tf32=True,
        max_grad_norm=0.3,
        warmup_ratio=0.03,
        lr_scheduler_type="constant",
        evaluation_strategy="epoch" if ft_parameters.DO_EVAL else "no",
        report_to="none",
        disable_tqdm=False,  # disable tqdm since with packing values are in correct
        full_determinism=True,
    )

Then when I load a model for merging I do the following I load it in bf16 as I thought it could only help - but your point is that it could deteriorate performance?

model = AutoPeftModelForCausalLM.from_pretrained(
    lora_adapter_dir,
    device_map=device_map,
    torch_dtype=torch.bfloat16
    )

model = model.merge_and_unload()

So it you opinion adding “quantization_config=bnb_config,” to from_pretrained when merging could help? But then I will quantize as well adapter which was in higher precision so far…
And probably the best option is to take bigger machine and finetune without any quantization then we avoid double quantization? I tested that quantization at gguf conversion is needed because w/o it the model has endless inference times…