Data format for fine-tune base model

AKeigo
Fri, 07 Mar 2025 05:52:49 +0000

I want to fine-tune a base model for single purpose (write a story).

There’re two ways to design the data:

1. {prompt}<special_token>{story}

2. You are a creative writing assistant. Please write a story accoarding to the prompt. Here is the prompt: {prompt}<special_token>{story}

Which one should I pick and why?
(I did an experiment, and it shows #1 is better. But I still a little curious about this.)