Q:

ValueError exception from a floor-division operation on a DataFrame

I am doing arithmetic on columns of a DataFrame. I get a ValueError exception when the size of the (sub-)DataFrame on which I operate exceeds a threshold.
As an example, consider the DataFrame:
import numpy as np
import pandas as pd

random_list = list(np.random.randint(1, 30, 10000))
df = pd.DataFrame({"p": random_list, "q": random_list})

This operation works:
df[0:5000] // [2, 10] 

but this one fails:
df[0:5001] // [2, 10]

with the exception:
ValueError: operands could not be broadcast together with shapes (10002,) (2,) 

As soon as the slice generates a DataFrame of len greater than 5000, an exception will be thrown. The threshold depends on the number of columns. Considering instead a DataFrame generated from the dictionary:
{"p": random_list, "q": random_list, "r": random_list}

and a length-3 list for the floor division, the above operation will fail for any slice of length greater than 3334.
What is happening?

A:

The deeper case goes way over my head, but I can tell you why the error happens.
In pandas\core\computation\expressions.py Line 31, the following constant is declared:
_MIN_ELEMENTS = 10000.
If the operation you try to perform exceeds this amount of elements to be evaluated, it will try to use the numexpr module to speed up computations. And while the floor division operator is supported in numexpr, the problem is that the expression is made into a VariableNode to be used in the expression, for which floor division is not supported. Keep in mind that for instance the addition operator is supported.
It stems from the following few lines:
if sys.version_info[0] < 3:
    __div__ = div_op
    __rdiv__ = binop('div', reversed=True)

In numexpr/expressions.py. The fun part is, that __div__ and __rdiv__ are never defined for sys.version_info[0] >= 3. If you add lines to implement these manually, your issue is solved. In other words: it is a bug in the numexpr module.
You can of course also declare _MIN_ELEMENTS = 20000 or similar, to expand the limit before you run into the same error.
A word of caution: you really want to be careful with editing parts of these kinds of core modules, as fiddling around with them can break a lot of stuff.

