About Pretrain

[AnjLiHe]

What should I do if I need to train MMM from scratch? I don't seem to find the pre-training code, only the fine-tuning code.

[victorywys]

We do not have the plan to open-source the code for pretraining stage, but you can try to do so by

1. Add MMMDecoder. We used a symmetry structure of encoder as the decoder.
2. For loss calculation, use representations from hypernodes to reconstruct input features. - At the time we trained the model, we reconstructed the DE features. But we're now aware of the issue maybe caused by it. You can try to modify it as the raw features. And we used the MSELoss.
3. Add a pretraining pipeline and feed all the available data as pretraining data.

Or if you're working with DE feature, you can use our pretrained checkpoint.