#logstash

Hi. So I've got a LS setup that does various filters and outputs to a few different places. How do I ensure that it keeps going even if one of the external outputs blocks up?

I have it archiving to local log files (MUST keep working), but also sending a copy to Loggly.org (non critical) and notifications to Nagios or Hipchat

at the moment if there's any kind of problem with one of the outputs, the whole thing backs up and stops.

ideas?

two instances with a rabbitmq with them seems recommended, but that feels like a large resource wasting solution and still doesn't solve keeping the other outputs working when one stops

at the moment if we have a minor network hiccough, it stops logging! and we can't diagnose

Heya

MugginsM that's the way it's designed, it'll change moving foward. you're best best is to break things up into chunks and then use a broker

warkolm: a LS instance for each output?

(with a decent queue in front)

just have the LS parser spit things back out into a broker, then run another LS instance for all your outputs to each endpoint

I am getting "Permission denied" messages in my logstash log, should I somehow start Logstash as root or is there a better way of dealing with this?

permission denied for what

warkolm: well, tried my best for tonight still getting parse errors, wish I could find out what on. If you happen to recall who wrote that article on sage, I'd be interested in talking with them. Thanks for help earlier. g'night

sid11 @bcshort on twitter is the person

Great, thanks!

what's a good value for a logstash java heap size? It seems java keeps on running out of memory when running logstash every 7-8 hours.

it shouldn't, what version are you on

1.4.2

I have the heap size set at 1g at the moment

that's pretty low, I'd go 4 personally

also upgrade to 1.4.4 as there will be bug fixes

i have a log source with a host field that seems to be overwritten by a host field that LS or LSF is replacing, anyone else had this issue?