The difference between Keras SGD and tf.train.MomentumOptimizer

[djshen]

There was a discussion about whether we can replace the tf.compat.v1.train.MomentumOptimizer with tf.keras.optimizers.SGD. To find the difference, I read the document and trace the source code.

The update rule of SGD is:
v_t+1 ← αv_t - η∇f(θ_t)
θ_t+1 ← θ_t + v_t+1

And that of MomentumOptimizer is:
v_t+1 ← αv_t - ∇f(θ_t)
θ_t+1 ← θ_t + ηv_t+1

The difference is that the learning rate is multiplied at the first step in SGD while it is multiplied at the second step in MomentumOptimizer.

If the learning rate η is a constant, the two formula are mathematically equivalent but there might be some floating point errors. However, if the learning rate is changing, the results will be different. Hope this could answer the question.

BTW, I find that the update rule in the slides is the same as SGD instead of MomentumOptimizer.