_selekt_and_load_pretrained and _selekt_and_load_previous, which is better ?

[shiwanghua]

by the way, here misses one parameter:

https://github.com/microsoft/NextCoder/blob/20168028f3d6f0b1f7059ef03ae72c7b7d968ff0/src/train/SeleKT/selekt.py#L231C9-L231C15

[SwayamInSync]

From the experiment we found load_pretrained is far better as it provides a static target, whereas in load_previous the accumulated scaled gradient comparison is happen against a dynamic moving target.