Concerns and Feedback on the HellaSwag Benchmark

[JhonDan1999]

Thank you for your effort for providing this benchmark

I have some concerns for this benchmark I want to share

While searching for examples for few-shot learning in the training set, I found it very difficult to locate examples that made sense. Many examples continuations do not make logical sense (leave alone grammatical and spelling errors).

The correct labels often seemed incorrect to me. This was apparent even with a brief review of the dataset; it was challenging to find correct examples, but easy to spot numerous nonsensical ones. Due to this, I believe it is unnecessary to provide specific examples here, as anyone who skims through the training set will likely encounter these issues.

Additionally, I found a blog post by a researcher that aligns closely with my observations and provides detailed examples.

Given the prominence of the Allen Institute in commonsense reasoning research, and your team’s reputation for excellence, I am reaching out to seek your insights and any justifications for these issues.
Given that HellaSwag is a well-known benchmark, this is quite concerning.

I would appreciate your perspective on this
thank you