DPO - Metric Interpretation

Intermediate
1

Hello, I’m new to DPO.

I’m currently working with DPOConfig(). During the training of my model, a few metrics are plotted such as “rewards/chosen", “rewards/rejected", “train/logps/rejected" etc.

While training, I see that the value for rewards/chosen goes up to 15 and rewards/rejected goes down to –35. What I don’t understand is what exactly is being plotted. What is the meaning of these numbers? They are not probabilities, so how should I interpret them?

3

Thank you for the reply. I have another question: when I choose SFT as the loss function in the DPOConfig, how is it still considered DPO, since SFT does not take rejected responses into account?