#logstash

someperson: Are you still around? I'm curious about https://github.com/elasticsearch/logstash/issue...

<http://tinyurl.com/n2fzqmx>; (at github.com)

_Bryan_: sup?

whack: is there anything I need to catch other than gc_logs that might tell me why my indexers are dieing north of 70k/s?

I am seeing this problem alot more since I added another 1000 shippers

and what do you write to logstash directly? redis?

rabbit?

im trying to use logstash with the embedded elasticsearch/kibana, but get this warning: WARN: org.elasticsearch.discovery: [logstash-logs-1118-4006] waited for 30s and no initial state was set by the discovery

im running logstash with this command: bin/logstash agent -f conf.conf web

and my conf.conf file looks like this: http://pastie.org/9511825

Title: #9511825 - Pastie (at pastie.org)

_Bryan_: you able to get a heap dump or stack trace?

I am watching them now....my site reliability team turned on handlers and was bringing it back up...

is there a way to tell java to not overwrite the gc log file on startup

I am runing these options with a 24GB heap

-Djava.io.tmpdir=${LS_HOME} -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/data-fio/ELK1/logs/gc_log.log

and 24 workers

OH..I can edit the init.d script to save log....I will go do that...

what java are you using?

Title: Private Paste - Pastie (at pastie.org)

this seems to be at many of the crashes

-XX:+PrintCompilation may be helpful as well

java-1.7.0-oracle-1.7.0.25-1jpp.1.el6_4.x86_64

paired with -XX:+TraceClassLoading and unloading

also -XX:-PrintConcurrentLocks

_Bryan_: dying how?

-XX:+TraceClassLoading -XX:+TraceClassUnloading

currently running but trying to set it up to get the info when it dies...

I expect it to crash this evening sometime..as I added alot of load to it about 2 hours ago

if not tonight when real production kicks up tomorrow

jlintz: is that right for the unloading option?

this is now java opts

-Djava.io.tmpdir=${LS_HOME} -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/data-fio/ELK1/logs/gc_log.log -XX:+PrintCompilation -XX:+TraceClassLoading -XX:-PrintConcurrentLocks

_Bryan_: ya, looks right, although that last one, should be a + instead of minus

for concurrent locks

k

I will maek the change on all 5 indexers

_Bryan_: my laptop can do 140k eps with a signle logstash agent.

_Bryan_: what error/crash/symptoms do you see?

whack: can i put your laptop into production here =)

in that link

Title: Private Paste - Pastie (at pastie.org)

that seems to always be there when it dies...

but I am setting up to catch them..have had issues since it overwrite the log file on restart...

I edited the init.d so when the handlers restart it the log will be saved

_Bryan_: what output is that? -verbose:gc?

what are the symptoms?

what makes you think it's GC?

when th eload gets steady for awhile over about 70k/s the logstash daemon basically stops its flow...and It starts processing only 10-20eps./.not thousands..jsut 10-20

I dont think it is GC..I dont know what it is..and I am looking for ways to get more information on it

look at cpu usage first

what outputs? what logstash config?

never over load of 6

24 cores..32GB ram..24GB heap...

let me grab config

can you show your config and logstash command line?

and if it gets stuck again, capture jstack and top -Hp <logstash_pid> output

yeah gimmie a sec

ok will add to notes for what to grab

do you need all filters?

will take me a few mins to redact if so

whack: http://pastie.org/private/uiwuvfg8kxbvqsyqazs6dq

Title: Private Paste - Pastie (at pastie.org)

that is most of the filters....I have removed others that are just json { source => "message" } and mutate to drop message

the commandline is at the bottom

flush_size => 50000 is probably not helping you

how big is your ES cluster?

given the behavior, this is probably ES not being able to keep up with the load

8 nodes running 2 ES instances each

on Fusio IO drives on the indexers are conencted via 10GB

is 192.168.150.11 a load balanacer?

primary master

don't do that.

there is a 3 moster cluster

don't send any requests to your master

if you have isolated master roles (very recommended), never send requests there

why 2 ES instances?

24gb heap is not recmomended for 32gb ram

_Bryan_: you'd benefit from support :P

ES has 146GB ram..and running 2 28GB heaps

indexrs have 23GB ram with 24GB heap