Introduction to Parallel Programming with MPI: Glossary

Many problems can be distributed across several processors and solved faster.

mpirun runs copies of the program.

The copies are distinguished by MPI rank.

Algorithms can have parallelisable and non-parallelisable sections.

A highly parallel algorithm may be slower on a single processor.

The theoretical maximum speed is determined by the serial sections.

The other main restriction is communication speed between the processes.

Use MPI_Send to send messages.

And MPI_Recv to receive them.

MPI_Recv will block the program until the message is received.

Two major paradigms, message passing and data parallel.

MPI implements the Message Passing paradigm.

Several standard patterns: Trivial, Queue, Master / Worker, Domain Decomposition, All-to-All.

Non-blocking functions allows interleaving communication and computation.

Use MPI_Barrier for global synchronisation.

All-to-All, One-to-All and All-to-One communications have efficient implementation in the library.

There are functions for global reductions. Don’t write your own.

Start from a working serial code.

Write a parallel implementation for each function or parallel region.

Connect the parallel regions with a minimal amount of communication.

Continuously compare with the working serial code.

Use a profiler to find the most important functions and concentrate on those.

The profiler needs to understand MPI. Your cluster probably has one.

If a lot of time is spent in communication, maybe it can be rearranged.

Start from a working serial code.

Write a parallel implementation for each function or parallel region.

Connect the parallel regions with a minimal amount of communication.

Continuously compare with the working serial code.