[Finetune] The fine-tuned result is so bad with Large amounts of data but better with less data， Anybody know why?

[ChristineSeven]

I finetuned Yi-6b with num_train_epochs=4 and 2 million pieces of high-quality data， the performance is so bad. But when I reduced the data to 20 thousands and num_train_epochs 4 , it performs ok. So anybody know why?

[findmyway]

It's hard to tell without looking into the details. I'd suggest you save checkpoints during training and watch some evaluation metrics.