Q:

Spark clean up shuffle spilled to disk

I have a looping operation which generates some RDDs, does repartition, then a aggregatebykey operation.  After the loop runs onces, it computes a final RDD, which is cached and checkpointed, and also used as the initial RDD for the next loop.
These RDDs are quite large and generate lots of intermediate shuffle blocks before arriving a the final RDD for every iteration. I am compressing my shuffles and allowing shuffles to spill to disk.
I notice on my worker machines that my working directory where the shuffle files are stores are not being cleaned up. Thus eventually I run out of disk space. I was under the impression that if I checkpoint my RDD, it would remove all the intermediate shuffle blocks. However this seems not to be happening. Would anyone have any ideas on how I could clean out my shuffle blocks after every iteration of the loop, or why my shuffle blocks aren't being cleaned up?

A:

Once you cached the RDD to your memory/disk, as long as the spark context is alive, the RDD will be stored in your memory/disk.
In order to tell the driver it can remove the RDD from the memory/disk you need to use the unpersist() function.

From the java-doc:
 /**
   * Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
   *
   * @param blocking Whether to block until all blocks are deleted.
   * @return This RDD.
   */
  def unpersist(blocking: Boolean = true)

So you can use:
rdd.unpersist()

