Results reproduction

[Xnhyacinth]

During the reproduction process, when I tested qwen-vl and the trained ckpts, I found that the mmvet results were all very low, less than 5. I wonder how the results in the paper were measured. Was the gpt_eval_score used? Also, for mathvista's testmini, the performance of the qwenvl base I tested was not as high as reported. Is it because I used vllm?

[Yangsenqiao]

Hi @Xnhyacinth 👋, thanks for your interest in VisionThink.

First, regarding MMVet: both the official Qwen checkpoint and our checkpoint also obtain low scores. I suggest checking the JSON file that contains your model outputs to verify the responses are correct and to see whether the code is misjudging them.

Second, for MathVista (testmini): we also use vLLM, but small performance gaps can be caused by environment differences. You can rerun the evaluation using the official lmms-eval codebase on the stable/v0d3 branch (the version from before 2025-07) for a cleaner comparison.

I’m looking into the issue as well.

[Xnhyacinth]

Hi, @Yangsenqiao , Thank you very much for your timely reply.

So how do I evaluate to calculate the similar scores of the papers? Because this is completely incomparable.

Thanks for the suggestions on MathVista (testmini).

[Yangsenqiao]

You might consider checking whether a similar issue exists in the lmms-eval library.

Also, for MMVet, could you please try the stable/v0d3 branch and see whether the results are still very low or if they look reasonable?

[Xnhyacinth]

Sure, I will use this branch to reproduce the results of the paper.

[Xnhyacinth]

@Yangsenqiao
Hi, is the score for MathVista (testmini) the average of the gpt_eval_scores of mathvista_testmini_cot, mathvista_testmini_format, and mathvista_testmini_solution?