How to Use MPI
Basics
This will give you a general overview of how to compile and execute a program that has been parallelized with MPI. Many of the options listed below are the same for both Open MPI and Intel MPI, however, be care if they do differentiate.
How to Compile MPI Code
Before continuing, please make sure that the openmpi or intelmpi module is loaded (go here to see how to load/switch modules).
There are several so called MPI "compiler wrappers", e.g. "mpicc". These take care of including the correct MPI libraries for each programming language etc. for you. But they share most command line options. Depending on whether your code is written in C, C++ or Fortran, follow the instructions in one of the tables below. Make sure to replace the arguments inside <…> with specific values.
Open MPI
Use the following command to specify the program you would like to compile (replace <src_file> with a path to your code, e. g. ./myprog.c).
| Language | Command |
| C | $ mpicc <src_file> -o <name_of_executable>
|
| C++ | $ mpicxx <src_file> -o <name_of_executable
|
| Fortran | $ mpifort <src_file> -o <name_of_executable>
|
You can also type the command $ mpicc [options], $ mpicxx [options] or $ mpifort [options]. There are a few options that come with Open MPI, however, options are more important for running your program. The compiler options might be useful to fetch more information about the Open MPI module you're using.
| Options | Function |
| -showme:help | print a short help message about the usage and lists all compiler options |
| -showme:version | show Open MPI version |
For RWTH cluster users:
Instead of typing the compiler wrapper mpicc etc., you can simply put one of the environment variables $MPICC, $MPICXX or $MPIFC for Fortran codes. They are already set by the module system so that you do not have to worry about which compiler module to use.
Intel MPI
Use the following command to specify the program you would like to compile (replace <src_file> with a path to your code, e. g. ./myprog.c).
| Compiler Driver | C | C++ | Fortran |
| GCC | $ mpicc <src_file> -o <name> |
$ mpicpc <src_file> -o <name> |
$ mpifort <src_file> -o <name>
|
| Intel | $ mpiicc <src_file> -o <name> |
$ mpiicpc <src_file> -o <name> |
$ mpiifort <src_file> -o <name>
|
You can also type the command $ mpicc [options] <src_file> -o <name> etc., where [options] can be replaced with one or more of the ones listed below. Intel MPI comes with rather advanced compiler options, that are mainly aimed at optimization and analyzing your code with the help of Intel tools.
| Options | Function |
| -g | enable debug mode |
| -O | enable compiler optimization |
| -v | print compiler version |
For RWTH cluster users:
Instead of typing the compiler wrapper mpicc etc., you can simply put one of the environment variables $MPICC, $MPICXX or $MPIFC for Fortran codes. They are already set by the module system so that you do not have to worry about which compiler module to use.
How to Run an MPI Executable
Ensure that the correct MPI module is loaded (go here to see how to load/switch modules). Once again, the command line options slightly differ between Intel MPI and Open MPI. In order to start any MPI program, type the following command where <executable> specifies the path to your application:
$ mpirun -n <num_procs> [options] <executable>
Note that mpiexec and mpirun are synonymous in Open MPI, in Intel MPI it's mpiexec.hydra and mpirun.
Don’t forget to put the “-np” or “-n” option as explained below. All the other options listed below are not mandatory.
Open MPI
| Option | Function |
| -np <num_procs> or -n <num_procs> | number of processes to run |
| -npersocket <num_procs> | number of processes per socket |
| -npernode <num_procs> | number of processes per node |
| -wdir <directory> | change to directory specified before executing the program |
| -nw | complete command when all MPI processes have been launched successfully |
| -path <path> | look for executables in the directory specified |
| -q or -quiet | suppress helpful messages |
| -output-filename <name> | redirect output into the file <name>.<rank> |
| -x <env_variable> | export the specified environment variable to the remote nodes where the program will be executed |
| --help | list all options available with an explanation |
Intel MPI
| Option | Function |
| -n <num_procs> | number of processes to run |
| -ppn <num_procs> | number of processes per node |
| -wdir <directory> | change to directory specified before executing the program |
| -path <path> | look for executables in the directory specified |
| -outfile-pattern <name> | redirect stdout to file |
| --help | list all options available with an explanation |
Options for Binding in Open MPI
Binding processes to certain processors can be done by specifying the options below when executing a program. This is a more advanced way of running an application and also requires knowledge about your system's architecture, e. g. how many cores there are (for an overview of your hardware topology, use $ lstopo). If none of these options are given, default values are set.
By overriding default values with the ones specified, you may be able to improve the performance of your application, if your system distributes them in a suboptimal way per default.
| Option | Function | Explanation |
| --bind-to <arg> | bind to the processors associated with hardware component; arg can be one of: none, hwthread, core, l1cache, l2cache, l3cache, socket, numa, board; default values: "core" for np <= 2, "socket" otherwise | e. g.: in case of "l3cache" the processes will be bound to those processors that share the same L3 cache |
| --bind-to-core | bind processes to cores | bind each process to a core |
| --bind-to-socket | bind processes to sockets | put each process onto a processor socket |
| --bind-to-none | bind no processes | do not bind any processes, but distribute them freely |
| --cpus-per-proc <num_cpus> | bind each process to the given number of CPUs | if set to 3, each process will take up 3 CPUs |
| --report-bindings | print any bindings for launched process to the console |