python - how to export a table dataframe in pyspark to csv? -


i using spark-1.3.1 (pyspark) , have generated table using sql query. have object dataframe. want export dataframe object (i have called "table") csv file can manipulate , plot columns. how export dataframe "table" csv file?

thanks!

if data frame fits in driver memory , want save local files system can convert spark dataframe local pandas dataframe using topandas method , use to_csv:

df.topandas().to_csv('mycsv.csv') 

otherwise can use spark-csv:

  • spark 1.3

    df.save('mycsv.csv', 'com.databricks.spark.csv') 
  • spark 1.4+

    df.write.format('com.databricks.spark.csv').save('mycsv.csv') 

in spark 2.0+ can use csv data source directly:

df.write.csv('mycsv.csv') 

Comments