about the performence

[Kenwwww]

I conducted tests on the following common rag situations and made the following findings:
1.There are several sentences in the long text that are related to the question
2. In long passages, one needs to have an overall understanding of the entire text to answer the question (that is, the knowledge points related to the question are very scattered).
3. Long texts have no relation to the question

The effect is as follows:

1. It can be found that there are no issues with the current test
2. The performance was poor. Instead of summarizing, it directly returned to the original text snippet
3. Poor performance, still returning the original text snippet

It seems that there is no summary process at all. It's more like "selecting sentences related to the problem from the original text".
But indeed, in terms of the performance of the task of selecting the relevant parts, it is indeed much better than the base model of the same scale.

[Levi-ZJY]

Thanks for your interest in our work and your question.

For our current training data, the answers are taken directly from the oracle documents. Thus, the model mostly learns to select sentences related to the question from the original text.

Regarding the setting where the knowledge points related to the question are scattered and the model needs to produce a coherent summary, which is a more challenging scenario, we don't have much training data for this, so the model tends to return original text snippets rather than a summary.

Thank you again for your valuable feedback.

[Kenwwww]

Thanks for your interest in our work and your question.

For our current training data, the answers are taken directly from the oracle documents. Thus, the model mostly learns to select sentences related to the question from the original text.

Regarding the setting where the knowledge points related to the question are scattered and the model needs to produce a coherent summary, which is a more challenging scenario, we don't have much training data for this, so the model tends to return original text snippets rather than a summary.

Thank you again for your valuable feedback.

I also found that when the query is relatively long (1k + length), the sentence filtering results of the model will repeat the sentences in the query (rather than filtering out the relevant sentences in the rag block).

From the perspective of model training, I have also encountered this problem before. Even the fine-tuning of models inevitably leads to a catastrophic collapse of the model's contextual ability and instruction compliance ability.

So I'd like to take the opportunity to ask if there are any suggestions regarding this topic? Whether it is the optimization of the above issues regarding this project, or the fine-tuning of the related adaptation to the disaster of collapse caused by specific tasks?