#logstash

another problem solved another problem to solve

sometimes I wonder if I would have been better off not learning anything at school and work as a shalf stacker :/

g0aliath: yes

g0aliath: 2 masters could lead to split brain

Reamer: i think that same thing every day, going to sell it all buy a RV and drive the world

g0aliath: we have 7 data nodes and 3 master nodes in our clusters

I'm trying to use gsub to remove the middle part of a log line which has been joined together using multiline (filter) but it doesn't seem to work how I would expect it to.... the regex should work, but it doesn't seemt oreplace anything, can anyone tell me why? http://pastebin.com/w7nF4kh5

Title: first gsub doesn't seem to have any effect - Pastebin.com (at pastebin.com)

jeffr76: thats correct, but I thought that if you used the formula it would be fine b/c you are declaring it should look for two masters

electrical I think I solved my issue by changing from UDP rsyslog to syslog, I think it has fixed the message order/multiline issue, but I guess I just have to wait and see :)

sorry changing to TCP rsyslog

Rumbles: hehe okay. do let me know :-)

Hi. We are trying to set up an ELK setup for logs. I am on this for 6 weeks with 3 colleagues and we cannot get the setup to perform the way we would need it to. We talked to some consulting people and they all just roll their usual "you need more hardware". We strongly thing the bottleneck is logstash but we cannot find the switch to flick to make it faster. Can someone recommend people who _know_ logstash and are willing to do consulting? Thanks.

Our current setup is: beaver as the log shipping tool, redis as the log buffer and elasticsearch as the "storage backend". with 7 instances of logstash on 7 hardware machines, we cannot drive the elasticsearch machine into i/o wait and the system only handles like 30k log lines per second.

kleind: you can look at the threads settings for inputs / ouputs and the -w flag for the filter threads.

We played with input threads for redis, output workers for elasticsearch and filter workers.

We have an extensive document of how these settings affected performance but nothing really pushes it past about 260 events per cpu core.

kleind: which LS version are you using? 1.4.2 i would assume?

electrical: yes

java 8

what is your current config threads, workers and filter workers wise?

4 input threads, 8 filter workers and 8 output workers seems to perform best in our situation.

machines range from 12 core/8gig machines to 32 core/64gig machines.

okay. remember that every thread / worker can utilize a full core.. so if you have more threads configured then cores they can mess things up.

In most cases the filters are the slowest. especially the grok filter

we played with this a _lot_. if we disable filtering, the throughput raises by about factor 2.5

kleind: okay.. hmmm

electrical: yeah, that's where i'm at

kleind: you could try out the 1.5.0 RC2 package and see if that yields any improvements for you?

electrical: i will try that.

in any case ... if anyone knows people who _know_ filtering in logstash, please let me know (via pn if you must)

kleind: what is your memory on the ls servers

kleind: and how are you sending to ES ?

jeffr76: memory ranges is 8g on 4 machines and 64g on 3 others. what exactly do you mean by "how are you sending to es?". gigabit ethernet if that's the question.

kleind: here is what mine looks like http://postimg.org/image/l3q95n2zr/

Title: View image: Logstash ES NEW 2 (at postimg.org)

does your LS servers also have ES installed

can you pastbin a config ?

jeffr76: installed yes, but not started. i currently have es only on one node. and this node is not saturated (ie no i/o wait and not all cpu cores in use)

on our environment we found that installing ES on the local LS nodes speed up the sending to ES

can you tell if there is a pipeline issue ?

in lS

ps. i was just dealing with this last night LOL

thanks for your help

bye

jeffr76: like how would i tell?

kleind: how is the CPU load on teh ls servers

i currently look at one ls on a 32 core machine. i see load 7 and logstash utilizes about 1200% cpu (ie 12 cores)

and whatever i configure in thraeds, workers etc. i cannot get it past that

i cannot get it to use all cpu cores

load=1min load avg

what os ?

debian wheezy

5g of mem for logstash. which should be fine, we looked at the jvm heap and that looked okay.

what are the workers and heap setto

4 input 8 filter and 8 output. heap 5gig.

we upped the numbers in various combinations to 25/25/25 and it does not get past the about 1200-1400% cpu usage

we are using -w 20 and LS_HEAP_SIZE=31257m

i can try that right now. give me a sec

but have and have 16 cores

becarefull with heap size if you only have 8G

i have noticed that more workers require more heap, even tho i did not see it used

the machine has 32 cores and 64gig of ram

o ok, go for i

s/i/ot/

OMG

will now try 4 input, 20 filter and 8 output with 32g heap now

with 4/8/8 and 5g i just saw about 5k events/s

that this machine wrote to es

i benchmark this with a redis dump of real life data

8million entries in there

last night we backed up 20M events

i was pissed

4/20/8 with 32g is at about 8.5k/s

on this one machine that is

better ?

from 5k to 8.5k

a little. still, the node is not saturated

still not great

what is your input

redis

whats your batch cout

default

we found that not to influence things

what ver of Redis

2.8

btw: that 8.5k/s still boils down to the previously mentioned 260 events/s per cpu core