c - Strange lag when sending large message using tcp_nodelay -

i'm writing c program sends output of bash shell on tcp connection. make program more responsive, used setsockopt() enable tcp_nodelay, disables nagle's buffering algorithm. worked great, except there lag in large messages. in, if message more around 500 bytes (probably 512). first 500 bytes go through (quickly in small messages), there'll 1-2 second delay before rest received @ once. happens once every 10-15 times large message received. on server side, message being written socket 1 byte @ time, , of bytes available, behavior unexpected me.

my best guess there's 512 byte buffer somewhere in socket that's causing block? did time tests see lag is, , i'm pretty sure it's socket lag occurring. of data on server side written without blocking, client receives end of message after lag. used getsockopt() find socket's receive , send buffers, , on 512 bytes - 66000 , 130000 respectively. on client side, i'm using express js receive data in handler (app.on('data', function(){})). read express function not buffer data?

would have guess why happening? thanks!

since tcp_nodelay means send every piece of data packet possible without combining data together, sounds sending tons of packets. since writing 1 byte @ time send packets 1 byte of payload , bigger frame. work fine of time first packet drops whatever reason receiver need go error-correction mode on tcp socket ask retransmission of dropped packet. incur @ least 1 round-trip latency , perhaps several. sounds getting lucky first several hundred packets (500 bytes worth) , typically hitting first packet drop , slowing way down due error correction. 1 simple solution might write in larger chunks, 10 bytes @ time, instead of 1 byte chance of hitting dropped packet less. expect see problem messages around 5000 bytes or so. in general setting tcp_nodelay cause things go faster @ first wind hitting first dropped packet sooner because tcp_nodelay not decrease number of packets send per amount of data. increases or leaves number of packets same means chance of hitting dropped packet within amount of data go up. there tradeoff here between interactive feel , first hiccup. avoiding tcp_nodelay can delay typical amount of data sent before first error retransmission hit on average.