Why can't I reproduce benchmark scores from papers like Phi, Llama, or Qwen? Am I doing something wrong or is this normal?

Pimpcat-AU
Tue, 10 Jun 2025 19:54:31 +0000

Yes, this is normal. Benchmark results in LLM papers are often not exactly reproducible due to differences in evaluation code, tokenizer versions, prompt formatting, dataset splits, hardware, random seeds, and sometimes undocumented “internal” settings or prompt engineering. Even “official” scripts can produce different results if any dependency changes.

Tips:

Use the exact tokenizer and prompt format as the original paper or repo.

Check for hidden preprocessing, special instructions, or test set modifications.

Use the same version of lm-eval-harness and its dependencies.

Run multiple seeds and average results if possible.

Even then, you may still see margin differences. This is a common and known issue in the LLM community.

Solution provided by Triskel Data Deterministic AI.