loss increasing

[FlyingAnt2018]

Hi, bro. I have just adjusted the epoch from 200 to 300, when training process going to epoch 100, all loss items have a significant increase. Do you know the reason ? Thanks.

[SrinjaySarkar]

Yeah I have observed the same exact thing. Could you find any reason @FlyingAnt2018 ?

[FlyingAnt2018]

Yeah I have observed the same exact thing. Could you find any reason @FlyingAnt2018 ?

lr too large

[SrinjaySarkar]

Could you provide the the lr you used to train it on 300 epochs ? @FlyingAnt2018

[2gunsu]

Hello. I'm sorry for the delay in answering. 😢
I'll check what you said and get back to you soon.

Thank you.

[sunhucheng]

Hello. I'm sorry for the delay in answering. 😢 I'll check what you said and get back to you soon.

Thank you.
I have observed the same problem, have you solved it? I think it's a matter of learning rate. @2gunsu

[sunhucheng]

I've found that this doesn't happen when the batch size is 8, but it does happen when it's 24. it can resume training from ckpt before loss incressing then loss will decreases.

[QSQSQSQS]

I have observed the same exact thing while training with nuscenes, but it didn't happen with KITTI.