MPI with SIMX

[rahul7rajdn]

MPI With SIMX

Usage

Apptainer> ./ci/blackbox.sh --cores=2 --app=mpi_vecadd --driver=simx --np=4 --args="-n5000"
CONFIGS=-DNUM_CORES=2
Running: CONFIGS="-DNUM_CORES=2" make -C ./ci/../runtime/simx > /dev/null
Running: OPTS="-n5000" NP=4 make -C "./ci/../tests/regression/mpi_vecadd" run-simx
make: Entering directory '/home/vortex/build/tests/regression/mpi_vecadd'
LD_LIBRARY_PATH=/home/vortex/build/runtime:/opt/boost-1.66/lib:/opt/openssl-1.1/lib::/.singularity.d/libs VORTEX_DRIVER=simx  mpirun  --allow-run-as-root --oversubscribe -np 4 ./mpi_vecadd -n5000
rank = 3, world_size = 4
rank = 0, world_size = 4
rank = 1, world_size = 4
rank = 2, world_size = 4
Rank: 3- Upload kernel binary
Rank: 0- Upload kernel binary
Rank: 1- Upload kernel binary
Rank: 2- Upload kernel binary
PERF: core0: instrs=22440, cycles=59003, IPC=0.380320
PERF: core1: instrs=22440, cycles=58635, IPC=0.382707
PERF: instrs=44880, cycles=59003, IPC=0.760639
PERF: core0: instrs=22440, cycles=59003, IPC=0.380320
PERF: core1: instrs=22440, cycles=58635, IPC=0.382707
PERF: instrs=44880, cycles=59003, IPC=0.760639
PERF: core0: instrs=22440, cycles=59003, IPC=0.380320
PERF: core1: instrs=22440, cycles=58635, IPC=0.382707
PERF: instrs=44880, cycles=59003, IPC=0.760639
PASSED!
PERF: core0: instrs=22440, cycles=59003, IPC=0.380320
PERF: core1: instrs=22440, cycles=58635, IPC=0.382707
PERF: instrs=44880, cycles=59003, IPC=0.760639
make: Leaving directory '/home/vortex/build/tests/regression/mpi_vecadd'
Apptainer> 

High-Level Summary of main.cpp

MPI Setup

Calls MPI_Init, gets the rank (MPI_Comm_rank) and world size (MPI_Comm_size).

Each MPI rank prints its rank and total world_size.

Argument Parsing

Reads -n from the command line (number of elements in the vector).

Rank 0 parses this value, then broadcasts it to all ranks with MPI_Bcast(&size, 1, MPI_UNSIGNED, 0, MPI_COMM_WORLD).

This ensures every rank sees the same problem size.

Data Partitioning

Total work = size elements.

Each rank computes its chunk:

  // Compute local chunk
  uint32_t chunk = (size + world_size - 1) / world_size; // ceil div
  uint32_t start = rank * chunk;
  uint32_t end   = std::min(start + chunk, size);
  uint32_t num_points = end - start;

So if size=50 and np=8, each rank gets about 6–7 elements.

Kernel Upload + Execution

Each rank loads the Vortex kernel binary (mpi_vecadd) into its own Vortex instance.

That’s why you see “Upload kernel binary” printed for every rank, not just once.

Then each rank launches the kernel for its assigned portion of the data.

Performance Reporting

After kernel finishes, each rank prints Vortex perf stats (instrs, cycles, IPC).

These numbers are per rank’s Vortex instance, not shared across ranks.

Verification

Each rank validates its results (checks that vector addition is correct).

Finally, the ranks synchronize (MPI_Barrier) and finalize (MPI_Finalize).

[Udit8348]

Hi Rahul, looks like a nice contribution to vortex simx 😄

I suspect that you tested locally on apptainer as reflected by your changes in vortex.def. To my knowledge we don't employ this file in the github actions (yet 👀 ) You will also need to update github actions pipeline to install the corresponding packages. It should be more-or-less similar to those installed in the apptainer def file.

I can tell this is failing the pipeline checks based on this failing test

Specifically if you expand Run Build you see the following

Run TOOLDIR=$PWD/tools
  TOOLDIR=$PWD/tools
  mkdir -p build32
  cd build32
  ../configure --tooldir=$TOOLDIR --xlen=32
  source ci/toolchain_env.sh
  make software -s > /dev/null
  make tests -s > /dev/null
  shell: /usr/bin/bash -e {0}
/bin/sh: 1: mpic++: not found

The simplest solution is to update ci/install_dependencies.sh with the corresponding packages.

As always there is potential to optimize this process in the future. One such idea is providing one source of ground truth between .def and workflows and docker etc. This takes some time, so focus on an minimal viable solution explained above.

Please, first test the full setup locally as best as you can (along with ci/install_dependencies.sh) rather than on github actions due to cost / time. Easiest way to isolate this issue is to take a clean ubuntu 22.04 env, update ci/install_dependencies.sh, run ci/install_dependencies.sh and check if mpi++ is available. Even better if you then complete a clean vortex install and build.