EAGLE-XL model crash with Intel compiler + Intel MPI

[bidabrain]

I am running EAGLE-XL model with Intel compiler + Intel MPI. The simulation started from a snapshot at z=4.
The code can initial proper and run for a while, but later when reach z=3.8. It crashed . I attached my library and compiler info setting as below. I have tried the exact same ICs, same SWIFT parameter file and using same MPI rands+ threads with the same SWIFT (2025.10) version but compiled with GCC12+OpenMPI4. Then the code can run without any issues. One big difference I noticed is that at the beginning after initialization, the Intel version SWIFT has a much better rank_CPU_balance than GCC+OpenMPI version, although they both use the same node/rank/threads number on the same machine. When the intel version crash, there is no error tracback info showed. only "===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 2 PID 2084916 RUNNING AT grammar067
= KILLED BY SIGNAL: 11 (Segmentation fault)

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 3 PID 2084917 RUNNING AT grammar067
= KILLED BY SIGNAL: 9 (Killed)

"
I am using Intel(R) oneAPI DPC++/C++ Compiler 2025.3.0

I would appreciate any guidance or insights you may have regarding this crash.”

Thanks

######################################################################

Welcome to the cosmological hydrodynamical code
______ _________________
/ / | / / _/ / /
_ | | /| / // // / / /
/ /| |/ |/ // // __/ / /
// |/|/_// //
SPH With Inter-dependent Fine-grained Tasking

Version : 2025.10
Revision: unknown, Branch: unknown, Date: unknown
Webpage : www.swiftsim.com

Config. options: '--with-hdf5=/home/dbi224/local/hdf5-1.14.6/bin/h5pcc --with-fftw=/home/dbi224/local/fftw3-intel --with-metis=/home/dbi224/local/METIS-intel --with-gsl=/home/dbi224/local/gsl-2.8 --with-parmetis=/home/dbi224/local/ParMETIS-intel --with-subgrid=EAGLE-XL --with-hydro=sphenix --with-kernel=wendland-C2'

Compiler: LLVM/Clang, Version: 21.1.1
CFLAGS : '-O2 -g -traceback -ffp-model=strict -ffp-contract=off -fno-fast-math -fno-alias -fprotect-parens -fimf-arch-consistency=true'

HDF5 library version : 1.14.6
FFTW library version : 3.x (details not available)
GSL library version : 2.8
MPI library version : Intel(R) MPI Library 2021.17 for Linux* OS (MPI std v4.1)
METIS library version : 5.2.1
ParMETIS library version : 4.0.3

[MatthieuSchaller]

Hi,

Could you clarify what simulation size this was? Also, can you show your runtime command line?
And what do you mean started from a snapshot at z=4?

[bidabrain]

Thanks for the quick response.

I am running a periodic boundary with 256^3 dark matter + 256^3 initial fluid elements in a 25 Mpc/h box. The initial condition file is generated by MUSIC2 at z=127. But since it used to crash at lower redshift. I am now testing with the snapshot at z=4 before crash as an initial condition file.
This is my computational node info :
NodeName=grammar001 Arch=x86_64 CoresPerSocket=32
CPUAlloc=0 CPUEfctv=64 CPUTot=64 CPULoad=0.00
AvailableFeatures=(null)
ActiveFeatures=(null)
Gres=(null)
NodeAddr=grammar001 NodeHostName=grammar001 Version=24.11.3
OS=Linux 5.14.0-570.17.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Fri May 23 22:47:01 UTC 2025
RealMemory=515697 AllocMem=0 FreeMem=336437 Sockets=2 Boards=1
State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
Partitions=normal,normal1
BootTime=2025-11-13T12:50:10 SlurmdStartTime=2025-11-13T15:23:13
LastBusyTime=2025-12-11T06:24:28 ResumeAfterTime=None
CfgTRES=cpu=64,mem=515697M,billing=64
AllocTRES=
CurrentWatts=0 AveWatts=0

When I submitted my slurm job, I asked for exclusive node for all the memory for the job. My HPC use UCX 1.18.0. IB Mellanox Technologies MT28908 Family [ConnectX-6]. When I test with GCC version SWIFT, it crash with some UCX errors so I fall back to TCP and then the GCC version SWIFT run without issue. So in my Intel version test I still stick on TCP. The runtime commond I am using as below.

export I_MPI_FABRICS=shm:tcp
export I_MPI_HYDRA_BOOTSTRAP=slurm

ulimit -v unlimited
ulimit -m unlimited
ulimit -d unlimited

mpirun -np 96 ./swift_mpi --threads=4 --pin
--cosmology --eagle --verbose=1
CV_23_restart.yml

[MatthieuSchaller]

Could you tell me more about the nodes you are running on? How many are you using and what is the memory / node?

Not that the code should crash but the way the setup is being run is quite unconventional. We would typically use only 1 MPI rank per node (or one per socket) and then use threads. This is always more efficient and will use less memory. That latter point could matter if things get (memory) imbalanced between the ranks.

[bidabrain]

The node I am running on has 2 Sockets per node and 32 AMD EPYC 7543 Cores per socket. The total physical memory is 512GB per node. I have experimented with different MPI/OpenMP layouts (nodes, ranks, and threads). For example, I using 8node X 12tasks/node X 4threads/task. It crashed within 15 mins. However, If I use 2node X 16tasks/node X 4threads/task, it can run around 3 hours (the simulation actually can reach lower redshift) then crash again. The errors are always same KILLED BY SIGNAL: 11 (Segmentation fault) and KILLED BY SIGNAL: 9 (Killed). Do you think if I should use higher threads number and lower MPI ranks per node may work better? Thanks.

[MatthieuSchaller]

Your calculation is small enough that it should fit into the memory of a single node.

So, it should work smoothly with just
./swift --threads=64 --cosmology --eagle --verbose=1 param_file.yml

If you go over the network, then yes, 1 rank / socket and 32 threads / rank would be my choide.
This will be faster. And use less memory.

Maybe one thing to worry about with your setup is whether SLURM and the mpi library properly distributed the ranks or whether they ended up in a bad setup (e.g. all on one node) which then leads to bad memory usage.
Note that different MPI libraries will do things differently.

[bidabrain]

I have tried only 1 Node x 2 Ranks x 32threads. Job running 1Hour10mins and crashed again. Only error info in the output log is
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 0 PID 3941733 RUNNING AT grammar064
= KILLED BY SIGNAL: 11 (Segmentation fault)

= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 1 PID 3941734 RUNNING AT grammar064
= KILLED BY SIGNAL: 9 (Killed)

in slurm error log, I also have some warning. However, I also see the same warning when I run with GCC compiled version SWIFT and it does not affect the code running at all. Do you have any other suggestions?

[0000] [00609.2] runner_do_limiter: WARNING: Not limiting particle with id 2953455 because it needs to be synced.
[0000] [00745.1] runner_do_limiter: WARNING: Not limiting particle with id 984209 because it needs to be synced.
[0001] [01044.9] runner_do_limiter: WARNING: Not limiting particle with id 24365339 because it needs to be synced.
[0000] [01193.3] runner_do_limiter: WARNING: Not limiting particle with id 4491433 because it needs to be synced.
[0000] [01193.3] runner_do_limiter: WARNING: Not limiting particle with id 334593 because it needs to be synced.
[0000] [01670.9] runner_do_limiter: WARNING: Not limiting particle with id 4391329 because it needs to be synced.
[0001] [01967.7] runner_do_limiter: WARNING: Not limiting particle with id 18244307 because it needs to be synced.
[0000] [01967.7] runner_do_limiter: WARNING: Not limiting particle with id 3211293 because it needs to be synced.
[0001] [01974.5] runner_do_limiter: WARNING: Not limiting particle with id 25954103 because it needs to be synced.
[0001] [02249.4] runner_do_limiter: WARNING: Not limiting particle with id 30749121 because it needs to be synced.

[MatthieuSchaller]

The warnings are not important. It's an internal thing which we should remove.

Could you run with -v 1 added to the command line?
And show me what the last messages are before the kill messages you get from the system?