Strong scaling performance, Collide benchmark, 64M particles
Performance in millions of particle-timesteps / second / node

Nodes SandyBridge Haswell Broadwell KNL K80-1 P100-1
1 61.48 (Kokkos/serial,mpi=16) 131.5 (Kokkos/serial,mpi=64,hyper=2) 142.6 (Kokkos/serial,mpi=72,hyper=2) 255.7 (Kokkos/serial/KNL,mpi=256,hyper=4) 139.7 (Kokkos/Cuda,mpi=2) 261.9 (Kokkos/Cuda,mpi=1)
2 64.67 (Kokkos/serial,mpi=16) 142.4 (Kokkos/serial,mpi=64,hyper=2) 155.4 (Kokkos/serial,mpi=72,hyper=2) 263.9 (Kokkos/serial/KNL,mpi=256,hyper=4) 125 (Kokkos/Cuda,mpi=2) 256.2 (Kokkos/Cuda,mpi=1)
4 69.67 (Kokkos/serial,mpi=16) 159.8 (Kokkos/serial,mpi=64,hyper=2) 175.3 (Kokkos/serial,mpi=72,hyper=2) 271.4 (Kokkos/serial/KNL,mpi=256,hyper=4) 129.5 (Kokkos/Cuda,mpi=2) 253.4 (Kokkos/Cuda,mpi=1)
8 80.31 (CPU,mpi=16) 184.8 (Kokkos/serial,mpi=64,hyper=2) 200.2 (CPU,mpi=72,hyper=2) 277.1 (Kokkos/KNL,mpi=128,thread=2,hyper=4) None None
16 95.94 (CPU,mpi=16) 211.7 (Kokkos/serial,mpi=64,hyper=2) 231 (Kokkos/serial,mpi=72,hyper=2) 287.9 (Kokkos/KNL,mpi=64,thread=4,hyper=4) None None
32 99.5 (CPU,mpi=16) 221.9 (CPU,mpi=64,hyper=2) 236.6 (Kokkos/serial,mpi=72,hyper=2) 287.6 (Kokkos/KNL,mpi=64,thread=4,hyper=4) None None
64 103.9 (CPU,mpi=16) 226.7 (CPU,mpi=64,hyper=2) 265.2 (CPU,mpi=36,hyper=1) 239.3 (Kokkos/KNL,mpi=32,thread=8,hyper=4) None None

Run commands and logfile links for column SandyBridge

1 mpirun -n 16 -N 16 --bind-to core spa_chama_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=chama.pkg=kokkos_serial.kind=strong.size=64M.node=1.mpi=16
2 mpirun -n 32 -N 16 --bind-to core spa_chama_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=chama.pkg=kokkos_serial.kind=strong.size=64M.node=2.mpi=16
4 mpirun -n 64 -N 16 --bind-to core spa_chama_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=chama.pkg=kokkos_serial.kind=strong.size=64M.node=4.mpi=16
8 mpirun -n 128 -N 16 --bind-to core spa_chama_cpu -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=chama.pkg=cpu.kind=strong.size=64M.node=8.mpi=16
16 mpirun -n 256 -N 16 --bind-to core spa_chama_cpu -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=chama.pkg=cpu.kind=strong.size=64M.node=16.mpi=16
32 mpirun -n 512 -N 16 --bind-to core spa_chama_cpu -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=chama.pkg=cpu.kind=strong.size=64M.node=32.mpi=16
64 mpirun -n 1024 -N 16 --bind-to core spa_chama_cpu -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=chama.pkg=cpu.kind=strong.size=64M.node=64.mpi=16

Run commands and logfile links for column Haswell

1 srun -n 64 -C haswell --ntasks-per-node 64 --cpu_bind=rank -c 1 ./spa_mutrino_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_serial.kind=strong.size=64M.node=1.mpi=64.hyper=2
2 srun -n 128 -C haswell --ntasks-per-node 64 --cpu_bind=rank -c 1 ./spa_mutrino_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_serial.kind=strong.size=64M.node=2.mpi=64.hyper=2
4 srun -n 256 -C haswell --ntasks-per-node 64 --cpu_bind=rank -c 1 ./spa_mutrino_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_serial.kind=strong.size=64M.node=4.mpi=64.hyper=2
8 srun -n 512 -C haswell --ntasks-per-node 64 --cpu_bind=rank -c 1 ./spa_mutrino_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_serial.kind=strong.size=64M.node=8.mpi=64.hyper=2
16 srun -n 1024 -C haswell --ntasks-per-node 64 --cpu_bind=rank -c 1 ./spa_mutrino_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_serial.kind=strong.size=64M.node=16.mpi=64.hyper=2
32 srun -n 2048 -C haswell --ntasks-per-node 64 --cpu_bind=rank -c 1 ./spa_mutrino_cpu -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=cpu.kind=strong.size=64M.node=32.mpi=64.hyper=2
64 srun -n 4096 -C haswell --ntasks-per-node 64 --cpu_bind=rank -c 1 ./spa_mutrino_cpu -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=cpu.kind=strong.size=64M.node=64.mpi=64.hyper=2

Run commands and logfile links for column Broadwell

1 mpiexec -np 72 -npernode 72 --oversubscribe --bind-to core ./spa_serrano_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=serrano.pkg=kokkos_serial.kind=strong.size=64M.node=1.mpi=72.hyper=2
2 mpiexec -np 144 -npernode 72 --oversubscribe --bind-to core ./spa_serrano_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=serrano.pkg=kokkos_serial.kind=strong.size=64M.node=2.mpi=72.hyper=2
4 mpiexec -np 288 -npernode 72 --oversubscribe --bind-to core ./spa_serrano_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=serrano.pkg=kokkos_serial.kind=strong.size=64M.node=4.mpi=72.hyper=2
8 mpiexec -np 576 -npernode 72 --oversubscribe --bind-to core ./spa_serrano_cpu -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=serrano.pkg=cpu.kind=strong.size=64M.node=8.mpi=72.hyper=2
16 mpiexec -np 1152 -npernode 72 --oversubscribe --bind-to core ./spa_serrano_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=serrano.pkg=kokkos_serial.kind=strong.size=64M.node=16.mpi=72.hyper=2
32 mpiexec -np 2304 -npernode 72 --oversubscribe --bind-to core ./spa_serrano_kokkos_serial -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=serrano.pkg=kokkos_serial.kind=strong.size=64M.node=32.mpi=72.hyper=2
64 mpiexec -np 2304 -npernode 36 --oversubscribe --bind-to core ./spa_serrano_cpu -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=serrano.pkg=cpu.kind=strong.size=64M.node=64.mpi=36.hyper=1

Run commands and logfile links for column KNL

1 srun -n 256 -C knl --ntasks-per-node 256 --cpu_bind=threads -c 1 ./spa_mutrino_kokkos_serial_knl -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_serial_knl.kind=strong.size=64M.node=1.mpi=256.hyper=4
2 srun -n 512 -C knl --ntasks-per-node 256 --cpu_bind=threads -c 1 ./spa_mutrino_kokkos_serial_knl -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_serial_knl.kind=strong.size=64M.node=2.mpi=256.hyper=4
4 srun -n 1024 -C knl --ntasks-per-node 256 --cpu_bind=threads -c 1 ./spa_mutrino_kokkos_serial_knl -sf kk -k on -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_serial_knl.kind=strong.size=64M.node=4.mpi=256.hyper=4
8 setenv OMP_NUM_THREADS 2; srun -n 1024 -C knl --ntasks-per-node 128 --cpu_bind=threads -c 2 ./spa_mutrino_kokkos_knl -sf kk -k on t 2 -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_knl.kind=strong.size=64M.node=8.mpi=128.thread=2.hyper=4
16 setenv OMP_NUM_THREADS 4; srun -n 1024 -C knl --ntasks-per-node 64 --cpu_bind=cores -c 4 ./spa_mutrino_kokkos_knl -sf kk -k on t 4 -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_knl.kind=strong.size=64M.node=16.mpi=64.thread=4.hyper=4
32 setenv OMP_NUM_THREADS 4; srun -n 2048 -C knl --ntasks-per-node 64 --cpu_bind=cores -c 4 ./spa_mutrino_kokkos_knl -sf kk -k on t 4 -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_knl.kind=strong.size=64M.node=32.mpi=64.thread=4.hyper=4
64 setenv OMP_NUM_THREADS 8; srun -n 2048 -C knl --ntasks-per-node 32 --cpu_bind=cores -c 8 ./spa_mutrino_kokkos_knl -sf kk -k on t 8 -pk kokkos reduction parallel/reduce comm classic -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.steps -log log.sparta.date=23Dec17.model=collide.machine=mutrino.pkg=kokkos_knl.kind=strong.size=64M.node=64.mpi=32.thread=8.hyper=4

Run commands and logfile links for column K80-1

1 mpirun -np 4 --npersocket 1 --bind-to core spa_ride80_kokkos_cuda -sf kk -k on g 2 -pk kokkos reduction atomic comm threaded -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.gpu.steps -log log.sparta.date=23Dec17.model=collide.machine=ride80.pkg=kokkos_cuda.kind=strong.size=64M.node=2.mpi=2.gpu=2
2 mpirun -np 8 --npersocket 1 --bind-to core spa_ride80_kokkos_cuda -sf kk -k on g 2 -pk kokkos reduction atomic comm threaded -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.gpu.steps -log log.sparta.date=23Dec17.model=collide.machine=ride80.pkg=kokkos_cuda.kind=strong.size=64M.node=4.mpi=2.gpu=2
4 mpirun -np 16 --npersocket 1 --bind-to core spa_ride80_kokkos_cuda -sf kk -k on g 2 -pk kokkos reduction atomic comm threaded -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.gpu.steps -log log.sparta.date=23Dec17.model=collide.machine=ride80.pkg=kokkos_cuda.kind=strong.size=64M.node=8.mpi=2.gpu=2
8 None
16 None
32 None
64 None

Run commands and logfile links for column P100-1

1 mpirun -np 2 --npernode 1 --bind-to core spa_ride100_kokkos_cuda -sf kk -k on g 1 -pk kokkos reduction atomic comm threaded -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.gpu.steps -log log.sparta.date=23Dec17.model=collide.machine=ride100.pkg=kokkos_cuda.kind=strong.size=64M.node=2.mpi=1.gpu=1
2 mpirun -np 4 --npernode 1 --bind-to core spa_ride100_kokkos_cuda -sf kk -k on g 1 -pk kokkos reduction atomic comm threaded -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.gpu.steps -log log.sparta.date=23Dec17.model=collide.machine=ride100.pkg=kokkos_cuda.kind=strong.size=64M.node=4.mpi=1.gpu=1
4 mpirun -np 8 --npernode 1 --bind-to core spa_ride100_kokkos_cuda -sf kk -k on g 1 -pk kokkos reduction atomic comm threaded -v x 256 -v y 160 -v z 160 -v t 100 -in in.collide.gpu.steps -log log.sparta.date=23Dec17.model=collide.machine=ride100.pkg=kokkos_cuda.kind=strong.size=64M.node=8.mpi=1.gpu=1
8 None
16 None
32 None
64 None