Q:

Unable to capture memory allocation failure

When I try to allocate a large memory using the 'new' operator in C++, system calls are not working after this memory allocation operation. It seems like the problem is with the memory allocation. However, I am not able to capture the exception. 
For better clarity, I have the following MWE.
My main.cpp contains the following lines:
#include <cstdio>
#include <iostream>
#include "memallocator.h"
#include <stdlib.h>

using namespace std;
const int MAX_ITER = 2000;

int main()
{
    printf("\n 1. output of ls: %d \n", system("ls"));

    for(int i=0;i<MAX_ITER;i++)
    {
        MemAllocator *pCMemAllocator;
        try
        {
            pCMemAllocator = new MemAllocator(); // alocates memory of size BUF_SIZE = 4000*1024;
        } catch(bad_alloc e)
        {
            printf("\nException at %d",i);
            break;
        }

    }
    printf("\nMemory Allocated");
    printf("\n 2. output of ls: %d \n", system("ls"));

    return 0;
}

The memallocator.h file is given as:
#ifndef MEMALLOCATOR_H
#define MEMALLOCATOR_H

const unsigned long BUF_SIZE = 4000*1024;

class MemAllocator
{
public:
    MemAllocator(){};
private:

    unsigned char ucBuffer [BUF_SIZE];

};

#endif // MEMALLOCATOR_H

I compiled it using g++ (version 4.8) on Ubuntu 14.04 64-bit and got the following output:
a.out  main.cpp  memallocator.h

 1. output of ls: 0 

Memory Allocated
 2. output of ls: -1 

That is, the system("ls") command fails after the memory allocation operation. However, the failure was not captured by the exception handler. I also tried the naive method: allocating zero first to the pointer and checking for zero after memory allocation operation. This method also did not capture the problem with the memory allocation.
If I set the variable MAX_ITER to some small value, say 20, everything works fine.
For higher values of MAX_ITER, the memory exception was supposedly handled by the exception handler. Why is it not happening?

A:

The system(3) function is (on Linux and most Posix systems) using the fork(2) syscall, which can fail for several reasons

EAGAIN: 
     fork() cannot allocate sufficient memory to copy the parent's
                page tables and allocate a task structure for the child.
ENOMEM: fork() failed to allocate the necessary kernel structures 
because memory is tight.

So you won't be able to  fork if your current process uses a lot of memory (and e.g. if the system does not have enough swap space to hold a copy).
BTW system(3) man pages says explicitly that system can fail and return....

   *  If a child process could not be created, or its status could not
      be retrieved, the return value is -1.

Consider using strace(1)  - proably as strace -f to understand the syscalls done by your application.
Notice that system(3) is not a syscall (which are listed in syscalls(2)...) but a standard library function.
So your huge memory allocation works as you want, but the later call to system is failing (and you should test it against failure). There is no memory allocation failure in your case.

