Q:

Python & Spark: Dataframe Write does not persist on filesystem

I want to write my dataframe to the local file system:
path = "/tmp/test"

df = spark.sql(select_str)
df.write.parquet(path, mode="overwrite", compression="snappy")

The code runs without an error but when I look into tmp there is no test And also If I want to read it back into pyarrow parquet:
parquet = pq.ParquetDataset(path)

I get the error:
IOError: Passed non-file path: /tmp/test

Same would happen for hdfs://tmp/test
What am I missing here?
EDIT
I tried creating a real file first before using it. So I created /tmp/parquet.parquet. The file can now be read but it is always empty. So the writing simply does not work.
EDIT 2
Spark Config:
spark = SparkSession.builder \
    .master("yarn") \
    .appName("my_app") \
    .config("spark.driver.bindAddress", "0.0.0.0") \
    .config("spark.driver.host", host) \
    .config("spark.driver.port", port) \
    .config("spark.driver.memory", "4G") \
    .config("spark.executor.memory", "8G") \
    .config("spark.blockManager.port", portBlockManager) \
    .enableHiveSupport() \
    .getOrCreate()

A:

In which mode are you running your spark application? Client/Cluster/Standalone?
Since you are trying to save file in local file system, you need to be aware about where actually your driver runs (as in which machine). 
If it is cluster mode, its very much possible that the file is getting written to some node on cluster where driver is created by resource manager. And your best way is storing file in hdfs, and by default that is what        df.write.parquet(path, mode="overwrite", compression="snappy") will do.
By default file system used will be hdfs, so if you simply provide /tmp/test it is looking for that path in hdfs and not local file system. 
Moreover I guess you are using pyarrow library in pq.ParquetDataset which by default will look for path on local file system. 
Making sure you use the correct file system during read/write will resolve it. 

