YugJ/Llama-3.2-3B-pricer-qlora

Llama-3.2-3B pricer (QLoRA)

Estimates the retail price of a product from its description.

Live results page: https://huggingface.co/spaces/YugJ/pricer-qlora Source code: https://github.com/Yugjohri/Finetuned-Open-Source-LLM-for-Product-Price-Prediction-QLoRA-

Llama-3.2-3B quantized to 4 bits with a rank-256 LoRA adapter, trained on a single 16 GB consumer GPU (RTX 5070 Ti). The reference recipe for this configuration asks for an A100 with high memory.

Results

Measured on 250 held-out rows, scored by the same harness. The base-model row is the control: it separates what the fine-tune taught from what Llama already knew about prices.

	Mean error	Median	R²	Within 20%
Base Llama-3.2-3B (4-bit)	$144.47	$40.80	−406.9%	12.4%
This adapter	$38.61	$17.99	73.1%	44.8%

The base model's negative R² means it scores worse than always guessing the average price. Across 250 products it gave only 27 distinct answers — $99.99 for 78 of them — reciting common price strings rather than reading the product.

Usage

The prompt format matters: the model learned exactly one shape, and a differently formatted prompt asks it a question it has never seen.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

BASE = "meta-llama/Llama-3.2-3B"
ADAPTER = "YugJ/Llama-3.2-3B-pricer-qlora"

quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=quant, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)

prompt = (
    "How much does this cost?\n\n"
    "Apple iPhone 15 Pro 128GB Smartphone\n"
    "Category: Electronics\n"
    "Brand: Apple\n"
    "6.1-inch Super Retina XDR display, A17 Pro chip, titanium design.\n\n"
    "Price is $"
)

ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=8, do_sample=False)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

pricer/prompt.py in the repository builds this string byte for byte.

Because the adapter is a separate set of matrices over the frozen base, you can compare against the untuned model without loading a second copy:

with model.disable_adapter():
    untuned = model.generate(**ids, max_new_tokens=8, do_sample=False)

Training

Base	meta-llama/Llama-3.2-3B, 4-bit NF4, double quant, bf16 compute
Data	ed-donner/items_prompts_full, 800,000 prompt/completion pairs, 1 epoch
LoRA	r=256, alpha=512, dropout 0.1, all 7 projections — 389 M trainable
Batch	4 × 64 accumulation = 256 effective
Optimiser	adamw_torch, lr 1e-4, cosine, weight decay 0.001, grad norm 0.3
Sequence length	128
Hardware	1 × RTX 5070 Ti (16 GB), peak 14.71 GB, 3,125 steps

Final eval loss 1.130, token accuracy 0.770.

Limitations

Trained on Amazon-style product listings, so it reflects those prices and the period the corpus covers. It cannot know about sales, regional pricing, or condition, and it quantizes toward round numbers — treat an estimate as a signal, not a quote. Accuracy degrades on expensive items: mean error is $6.78 under $25 but $159.60 above $300.

Credits

The corpus and the QLoRA recipe are from Ed Donner's LLM Engineering course. The single-GPU adaptation, evaluation harness and demo are by Yug Johri.

Built with Llama. Llama 3.2 is licensed under the Llama 3.2 Community License, © Meta Platforms, Inc.