Spaces:
Running on CPU Upgrade
Upload Taichu-Embedding-2B.json
Dear organizers,
We have update the evaluation results of our Taichu-Embedding-2B on the MMEB benchmark via a Pull Request and hope to have our model's new results included in the MMEB leaderboard.
In this PR, I submitted the results for both the 2B and 9B models (see: https://huggingface.co/spaces/TIGER-Lab/MMEB-Leaderboard/discussions/154).
However, the 2B model results failed to show up on the MMEB-Image leaderboard, while the 9B model results displayed normally. I’m unsure what caused this issue, so I resubmitted the 2B results separately as a new single submission.
Hi @gaoyuanzi , I can see both Taichu-Embedding 2B and 9B are already on the leaderboard (#7 &# 64 on Image), could you please double check that?
Thank you for your reply. However, there seems to be a bug. When I uploaded the Image results for both Taichu-Embedding 2B and 9B together, the metrics for Taichu-Embedding 2B were not updated correctly. On the Image leaderboard, the scores of the 2B model on the three test sets — RefCOCO, RefCOCO-Matching and Visual7W-Pointing — show as zero, whereas the values in the uploaded results are non-zero.
I’m not sure if the parsing error occurred because I included an extra hit@3 metric in the results. May I remove hit@3 and re-upload the data?
Hello @gaoyuanzi Having hit@3 should not be a problem, as long as your hit@1 exists. It seems that your 2b model is missing the three datasets's scores in the uploaded score sheet.
I also checked your most recent pull request and confirmed that the three datasets are indeed missing. Your screenshot seems to be the result of the 9B model.
Could you please double check your 2B model's score sheet, and let us know if it still doesn't work?