[MiniLLM] Why the teacher prob are not extract offline?

[huge123]

Thanks for sharing this great work.

But I have a question: As the title suggests, why do not we extract the teacher in advance to save memory and training time, as the input training prompts are loaded from a fixed dataset?

[t1101675]

In MiniLLM, we need the teacher model's probs on the student model's responses, which are generated online. Therefore, the teacher probs cannot be generated offline.

[huge123]

In MiniLLM, we need the teacher model's probs on the student model's responses, which are generated online. Therefore, the teacher probs cannot be generated offline.

I got it, I saw the teacher mix sampling, and the length normalization requires the sample from the student as input. Thanks for your reply.