Why can't I reproduce benchmark scores from papers like Phi, Llama, or Qwen? Am I doing something wrong or is this normal?

John6666
Wed, 28 May 2025 00:55:36 +0000

The backend, or rather, if the library version or options passed to the generation function (such as temperature) are different, the results may vary, so I think it can only be used as a rough guide. Leaderboards are easy to compare because they use the same criteria within the same leaderboard, but I don’t think there are many absolute indicators that can be used. For large companies, I think the output of the endpoints officially provided by the company can be used as a reference.

GitHub

LLM inference server performances comparison llama.cpp / TGI / vLLM ·...

Performances and improvment area This thread objective is to gather llama.cpp performance 📈 and improvement ideas💡against other popular LLM inference frameworks, especially on the CUDA backend. Let...

huggingface.co

Spaces - Hugging Face

Discover amazing ML apps made by the community