Q:

Failures with Python multiprocessing.Pool when maxtasksperchild is set

I am using Python 2.7.8 on Linux and am seeing a consistent failure in a program that uses multiprocessing.Pool(). When I set maxtasksperchild to None, then all is well, when testing across a variety of values for processes.  But if I set maxtasksperchild=n (n>=1), then I invariably end with an uncaught exception.  Here is the main block:
if __name__ == "__main__":
    options = parse_cmdline()
    subproc = Sub_process(options)

    lock = multiprocessing.Lock()
    [...]
    pool = multiprocessing.Pool(processes=options.processes,
                                maxtasksperchild=options.maxtasksperchild)
    imap_it = pool.imap(recluster_block, subproc.input_block_generator())
    #import pdb; pdb.set_trace()
    for count, result in enumerate(imap_it):
        print "Count = {}".format(count)
        if result is None or len(result) == 0:
            # presumably error was reported
            continue
        (interval, block_id, num_hpcs, num_final, retlist) = result
        for c in retlist:
            subproc.output_cluster(c, lock)
    print "About to close_outfile."
    subproc.close_outfile()
    print "About to close pool."
    pool.close()
    print "About to join pool."
    pool.join()

For debugging I have added a print statement showing the number of times through the loop.  Here are a couple runs:
$ $prog  --processes=2 --maxtasksperchild=2
Count = 0
Count = 1
Count = 2
Traceback (most recent call last):
  File "[...]reclustering.py", line 821, in <module>
    for count, result in enumerate(imap_it):
  File "[...]/lib/python2.7/multiprocessing/pool.py", line 659, in next
    raise value
TypeError: 'int' object is not callable

$ $prog --processes=2 --maxtasksperchild=1
Count = 0
Count = 1
Traceback (most recent call last):
[same message as above]

If I do not set maxtasksperchild, the program runs to completion successfully.  Also, if I uncomment the "import pdb; pdb.set_trace()" line and enter the debugger, then the problem does not appear (Heisenbug).  So, am I doing something wrong in the code here?  Are there conditions on the code that generates the input (subproc.input_block_generator) or the code that processes it (recluster_block), that are known to cause issues like this?  Thanks!

A:

maxtasksperchild causes multiprocessing to respawn child processes. The idea is to get rid of any cruft that is building up. The problem is, you can get new cruft from the parent. When the child respawns, it gets the current state of the parent process, which is different than the orignal spawn. You are doing your work in the script's global namespace, so you are changing the environment the child will see quite a bit. Specifically, you use a variable called 'count' that masks a previous 'from itertools import count' statement.
To fix this:

use namespaces (itertools.count, like you said in the comment) to reduce name collisions
do your work in a function so that local variables aren't propagated to the child.

