r - Having saved a dataframe to hdfs I have an error when I try to unserialize it when reading it back in using rhdfs -

i have written dataframe hdfs using rhdfs library , when try read in have errors.

the code write dataframe follows,

df.file <- hdfs.file("/mydir/df.rdata", "w") hdfs.write(df, df.file) hdfs.close(df.file)

and read in use

df.file <- hdfs.file("/mydir/df.rdata", "r") m <- hdfs.read(df.file) df <- unserialize(m) hdfs.close(df.file)

but error @ unserialize stage,

error in unserialize(m) : read error

does have idea cause of error , can prevent it. appreciated.

this happens when object unserialize bigger 65536 bytes

if @ rstudio environment, see df object raw[1:65536] , missed part of file

you should read pieces code:

WIKI