HDF5 output *very* slow on multiple nodes.

[connorourke]

I am testing a build of mpb on a HPC system, but having issues with outputting bands on multiple nodes.

If I run on multiple nodes the output of the bands seems very slow - around 2/3 .h5 files of ~110 MB per minute. But if I run on a single node it is significantly faster (while the algorithm itself, obvisouy, runs more slowly).

Is this expected - and what can be done to tune the HDF5 output when running over multiple nodes? I have tried several different toolchains/HDF5 builds and all seem to experience this problem.

Any advice on getting a performant build would be appreciated!

[stevengj]

Is MPB compiled/linked with a parallel/MPI build of HDF5?

If you link to a serial version of HDF5, then it will be much slower (it sends chunks to the master node one at a time to write out, IIRC).

[connorourke]

Yeah @stevengj - the hdf5 is built with MPI.

And I think I checked the HDF5 lib was performant over multiple nodes. (Been a while since I built it)

[connorourke]

OK - I just checked and the HDF5 seems to have a similar problem with a stand alone code that does a collective write on a single file when run over multiple nodes.

I guess this is a problem with the HDf5 install then - as the output from MPB with multiple nodes should be faster than with a single node?

i.e. I should expect the MPB output with multiple nodes to be significantly faster than it is, i.e. the significant slow-down i see when switching to multiple nodes doesnt happen for others?

btw I'm using:

mpb_mpi 1.12.0, Copyright (C) 1999-2012, MIT
Using libctl 4.5.1 and Guile 2.2.7.