Evaluation loss depends on batch size

[John6666]

Apparently a known but neglected issue for several years…

Batch size during training vs batch size during evaluation

I am confused about the difference between batch size during training versus batch size during evaluation. I am trying to measure how batch size influences the inference time(speed of prediction) of different NLP models after they have been trained using the Huggingface Trainer API. The code I used is below def print_summary(result): print(f"Time: {result.metrics['test_runtime']:.2f}") print(f"Samples/second: {result.metrics['test_samples_per_second']:.2f}") print(f"Test Loss: {res…