ABSTRACT: We have recently performed the largest 3D global magnetosphere hybrid simulation to date using 98,304 cores on NSF’s XT5 supercomputer Kraken. The unprecedented simulations have been instrumental in revealing new science which has now been confirmed in spacecraft data. This work highlights the challenges of running simulations at large core counts. One significant issue is the large I/O requirements for particle based codes. Particle codes can produce more than 200TB in a single run and will soon scale to 10s of PBs. We present Lustre tuned optimizations that resulted in a factor of 8 speed up in our IO kernel, making our unprecedented simulation possible. We present a scaling study and investigation of Lustre specific tuning for POSIX, independent MPI-IO and collective MPI-IO making use of the new Lustre specific lock protocol based file domain partitioning scheme implemented in Cray MPT.
Homa Karimabadi - University of California, San Diego
Burlen Loring - University of California, San Diego