Q:

Simple way to reverse dask distributed dataframe

I tried to reverse the order of a dask dataframe using [::-1], but got a NotImplementedError that you can only use iloc indexing like [:, ['foo']]
e.g
tmp=pd.DataFrame(dict(a=[0,1,1,1,0,1,0,1], b=[0,0,0,0,1,0,0,1]))
tmp=dd.from_pandas(tmp, npartitions=4)
tmp[::-1]

How can I easily reverse the order of a sorted dataframe, without loading the entire dataframe in memory?

A:

Here is a solution that leaves the index unchanged:
@dask.delayed
def reverse_pdf(pdf):
    '''delayed function to reverse a pandas dataframe'''
    return pdf[::-1]

# generating testdata
tmp=pd.DataFrame(dict(a=[0,1,1,1,0,1,0,1], b=[0,0,0,0,1,0,0,1]))
tmp_dd=dd.from_pandas(tmp, npartitions=4)

# reversing tmp_dd
ds = tmp_dd.to_delayed() # one delayed object per partition
ds = [reverse_pdf(d) for d in ds] # reverse each partition
ds = reversed(ds) # reverse the order of the partitions
tmp_dd_reversed = dd.from_delayed(ds) # construct a new dask dataframe

