SausageLM-7b-Instruct-v0.01-dpo-qlora

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.2 on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

• Loss: 0.4204
• Rewards/chosen: -1.9644
• Rewards/rejected: -3.5978
• Rewards/accuracies: 0.8020
• Rewards/margins: 1.6333
• Logps/rejected: -778.7791
• Logps/chosen: -552.1046
• Logits/rejected: 1.3639
• Logits/chosen: 0.3998

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.4906	0.08	300	0.5340	-1.1814	-1.8425	0.7310	0.6611	-603.2533	-473.8014	-1.6234	-1.7536
0.4794	0.16	600	0.4701	-1.3882	-2.4799	0.7700	1.0918	-666.9945	-494.4773	1.2460	0.4450
0.4519	0.24	900	0.4566	-1.4239	-2.6724	0.7730	1.2485	-686.2431	-498.0537	1.0803	0.1979
0.4034	0.31	1200	0.4487	-1.9028	-3.5170	0.7870	1.6142	-770.7061	-545.9451	1.7156	0.7244
0.4193	0.39	1500	0.4420	-1.8864	-3.4847	0.7840	1.5983	-767.4712	-544.3021	0.9998	0.0019
0.409	0.47	1800	0.4365	-2.0591	-3.7221	0.7920	1.6630	-791.2130	-561.5723	1.4876	0.5341
0.4037	0.55	2100	0.4334	-2.1275	-3.8835	0.7970	1.7560	-807.3529	-568.4110	1.9485	0.9489
0.3829	0.63	2400	0.4248	-1.8791	-3.4902	0.8010	1.6111	-768.0193	-543.5670	1.5421	0.5047
0.47	0.71	2700	0.4211	-1.8565	-3.4027	0.8030	1.5462	-759.2699	-541.3088	1.5152	0.5343
0.3769	0.79	3000	0.4205	-1.9199	-3.5317	0.8010	1.6119	-772.1762	-547.6463	1.5142	0.5326
0.3921	0.86	3300	0.4216	-2.0430	-3.7240	0.8050	1.6810	-791.3992	-559.9616	1.5287	0.5531
0.4249	0.94	3600	0.4204	-1.9591	-3.5883	0.8000	1.6292	-777.8283	-551.5704	1.3533	0.3917

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.15.0