Data format for fine-tune base model

AKeigo
Mon, 10 Mar 2025 01:08:18 +0000

Thanks for helping

The experiment I ran has some bug, so I did some other tests.

Here’s my obersavation and guesses:

1. I fine-tune Llama-3.1-base 8B on 10k dataset.

2. On the new experiment, #2 starts with a litter smaller loss, but at the end of training, #1 and #2 has almost the same loss.

3. So, I think both formats is OK. Personally, I think #1 is more elegant. But if you have a brilliant system prompt, or don’t have enough data, then #2 is worth to try.