Could you consider releasing the ground-truth answers for the program understanding tasks?

[gaohongwan]

Hi there,

First of all, thank you so much for providing such well-designed program understanding tasks! We find them truly valuable for evaluating our models’ reasoning and comprehension capabilities in code-related scenarios.

However, one thing we noticed is that the ground-truth answers are not currently available, which makes it a bit difficult for us to conduct accurate evaluations in our experiments.

We were wondering:

Would you consider releasing the answers?
Having access to them would greatly help us benchmark and analyze performance more effectively.

Thanks again for your amazing work and openness — we’re looking forward to seeing more progress on this project!

Warm regards,
Email: gaohongwan@bytedance.com