i have huge json file, small part follows:
{ "socialnews": [{ "admintagids": "", "fileids": "", "departmenttagids": "", ........ ........ "comments": [{ "commentid": "", "newsid": "", "entityid": "", .... .... }] }] ..... } i have applied lateral view exlode on socialnews follows:
val rdd = sqlcontext.jsonfile("file:///home/ashish/test") rdd.registertemptable("social") val result = sqlcontext.sql("select * social lateral view explode(socialnews) social comment") now want convert result (dataframe) json , save file, not able find scala api conversion. there standard library or way figure out?
val result: dataframe = sqlcontext.read.json(path) result.write.json("/yourpath") the method write in class dataframewriter , should accessible on dataframe objects. make sure rdd of type dataframe , not of deprecated type schemardd. can explicitly provide type definition val data: dataframe or cast dataframe todf().
Comments
Post a Comment