bash: Convert human readable time to epoch in large files -

i have huge csv files (>50 gb) have following columns:-

"time", "data column"

the time column has data "2015-07-10 00:17:45.000 gmt". wish convert time column reflect equivalent epoch time instead of human readable form.

what can fastest way without creating file? (this csv file output of python script).

date --date="2015-07-10 00:17:45.000 gmt" '+%s' 

will print

1436487465 

you can use cut rest:

#!/bin/bash while ifs='' read -r line || [[ -n $line ]];   timefield=$(echo "$line" | cut -f1 -d, | tr -d '"')   epochal=$(date --date="$timefield" '+%s')   restofline=$(echo "$line" | cut -f2- -d,)   echo "$epochal,$restofline" done <$1 

to use above script save file, chmod executable, , run supplying filename read first argument.

so inp.tst:

"2015-07-10 00:17:45.000 gmt","misc data","blah" "2015-07-10 00:18:45.000 gmt","more data","misc" 

you can use:

./fixtimes.sh inp.tst  

to get

1436487465,"misc data","blah" 1436487525,"more data","misc"