my inputs hundreds of big 1-line json files (~10mb-20mb).
after getting out-of-memory errors real setup (with 2 custom filters), simplified setup isolate problem.
logstash --verbose -e 'input { tcp { port => 5000 } } output { file { path => "/dev/null" } }' `
my test input multi-level nested object in json:
$ ls -sh example_fixed.json 9.7m example_fixed.json if send file once, works fine. if do:
$ repeat 50 cat example_fixed.json|nc -v localhost 5000 i error message:
logstash startup completed using version 0.1.x codec plugin 'line'. plugin isn't supported community , has no maintainer. {:level=>:info} opening file {:path=>"/dev/null", :level=>:info} starting stale files cleanup cycle {:files=>{"/dev/null"=>#<iowriter:0x6f51765 @active=true, @io=#<file:/dev/null>>}, :level=>:info} error: application used more memory safety cap of 500m. specify -j-xmx####m increase (#### = cap size in mb). specify -w full outofmemoryerror stack trace i have determined error triggers if send input more 30 times heap size of 500mb. if increase heap size, limit goes accordingly.
however, documentation understand logstash should able throttle input when cannot process events enough.
in fact, if sleep 0.1 after sending new events, can handle 100 repetitions. not 1000. assume means input not being throttled properly, , whenever input rate higher processing rate, it's matter of time before heap filled , logstash crashes.
Comments
Post a Comment