The implementation of SN can synchronize moving_mean and moving_variance across multi GPUs.
from switch_norm import switch_norm
if you use tf.slim, just replace slim.batch_norm with switch_norm. They share alomost the same parameters.
neuleaf/Switchable_Normalization_TF
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|