Validation file is loaded but no per-epoch validation appears in the training log

[LiloYJL]

Hello everyone,
I’m a new user of MACE and I’m encountering an issue with validation output during training. My mace version is 0.3.14.
In my command, I explicitly set:
--train_file=TRAINING_RANDOM.xyz
--valid_file=VALIDATION.xyz
--test_file=TEST.xyz
--eval_interval=1
In the training log, I can see that:
The validation dataset is successfully loaded
At the end of training, the final validation RMSE is printed
However, the problem is:
During training, I do not see any per-epoch validation results
Throughout the entire training process, only the training loss and training RMSE values are printed, e.g.:
Epoch 0: loss=..., RMSE_E=..., RMSE_F=...
Epoch 1: loss=..., RMSE_E=..., RMSE_F=...
...
But no validation loss / RMSE is shown, even though I have explicitly set --eval_interval=1.
I have already confirmed that:
The training / validation / test files exist
All datasets are loaded correctly (the log shows validation is combined successfully)
My questions
Why are validation results not printed at each epoch?
Is there something wrong in my script?
Or does the latest MACE version suppress per-epoch validation output by default?
I need validation RMSE for energy / forces / stress for every epoch
so that I can plot learning curves and identify the point where overfitting begins.
How can I properly enable per-epoch validation logging?
Additional context
As far as I know, to determine the “optimal non-overfitting point,” I need to plot:
Training RMSE (E/F/stress)
Validation RMSE (E/F/stress)
Versus epoch
Right now, I only have the training curves, and no validation data at all during training, which makes it impossible to evaluate overfitting.
Thank you very much for any help!
I can also provide my full run script and log file if needed.

[ilyes319]

The metrics printed in the logs are only validation metrics and not training metrics. The training metrics are not aggregated, and one should only monitor the validation metric. You can find the per-batch training loss in the results folder in the _train.txt file.

Note that overfitting in modern neural networks is not following the same trends, and looking at the training loss is not a good indication of that. One should only look if the validation error increases or plateaus, which is automatically handled by the patience.

Finally, we already plot the validation curves at the end of the training; they are stored in the results folder too.

[LiloYJL]

Thanks for your quick reply. I have a few additional questions regarding the training and validation metrics in MACE.

1. From your explanation, I understand that the log file only shows the validation metrics, while the _train.txt file stores the training metrics. Regarding overfitting, MACE will automatically handle it using the patience mechanism, so I don’t need to manually determine whether the model is overfitting — is that correct?
2. I only found the _train.txt file in the results folder, but I didn’t see any validation curves. Do I need to add any specific command in my training script to generate those plots?
3. If I understand correctly, in the _train.txt file:
   Entries with mode = "opt" correspond to training metrics (i.e., metrics calculated on the training set during each optimization step).
   Entries with mode = "eval" correspond to validation metrics.
   However, in my _train.txt file, I only see eval entries at the first epoch and the last epoch. There are no validation metrics recorded for intermediate epochs. I wonder if MACE only evaluates the validation set at certain checkpoints by default, or if I need to add any specific command to record validation metrics at every epoch.

[ilyes319]

1. That is correct. Be on the look out if the validation starts to increase after a while, but mace will automatically save the model with the best validation anyway.
2. What version of MACE are you using? It should save the plot at the final stage after training. If you want to plot the metrics every N steps, use --plot_frequency=N \ in your input.
3. Well there should be. MACE evaluates the valid set every N steps that you can control with --eval_interval=N , by default it is 1.

[LiloYJL]

My MACE is 0.3.14. I will try the way you advised. Thanks!