Q:

MPI - Use multiple threads to listen for incoming messages

I am working on a project that uses MPI routines and multiple threads for sending and receiving messages. I would like each receiving thread to focus on a different incoming message instead of having two or more trying to receive the same one. Is there a way to achieve this?
I don't know if this helps but I am currently using Iprobe() to check for incoming messages and Irecv() with Test() to check if the thread has received the whole message.

A:

Starting with version 3 of the standard, MPI allows for the removal of matched messages from the message queue so that they are no longer visible to subsequent probes/receives. This is done using the so-called matched probes. Just replace MPI_Iprobe with MPI_Improbe, which is the non-blocking matched probe operation:
int flag;
MPI_Status status;
MPI_Message msg;

MPI_Improbe(source, tag, comm, &flag, &msg, &status);

Once MPI_Improbe returns 1 in flag, a message matching (source, tag, comm) has arrived. A handle to the message is stored into msg and the message is removed from the queue. Subsequent probes or receives with a matching (source, tag, comm) triplet - by the same thread or in another - won't see the same message again and therefore won't interfere with its reception by the thread that matched it originally.
To receive a matched message, use MPI_Imrecv (or the blocking MPI_Mrecv):
MPI_Request req;
MPI_Imrecv(buffer, count, dtype, &msg, &req);
do
{
   ...
   MPI_Test(&req, &flag, &status);
}
while (!flag);

Versions of MPI before 3.0 do not provide similar functionality. But, if I understand you correctly, you only need to guarantee that no matching probe will be posted before MPI_Irecv has had the opportunity to remove the message from the queue (which is what matched probe+receive is meant to prevent). If you are probing in a master thread and then dispatching the messages to different threads, then you could use a semaphore to delay the execution of the next probe by the main thread until after the worker has issued MPI_Irecv. If you have multiple threads doing probe+receive, then you may simply issue the MPI_Irecv call in the same critical section (or whatever synchronisation primitive you use to achieve the serialisation of the MPI calls as required by MPI_THREAD_SERIALIZED) as MPI_Iprobe once the probe turns out successful:
// Worker thread
CRITICAL(mpi)
{
  MPI_Iprobe(source, tag, comm, &flag, &status);
  if (flag)
    MPI_Irecv(buffer, count, dtype, status.MPI_SOURCE, status.MPI_TAG, comm, &req);
}

Replace the CRITICAL(name) { ... } notation with whatever primitives your programming environment provides.

