#logstash

i followed this tutorial: https://www.digitalocean.com/community/tutorial...

<http://tinyurl.com/q8r9363>; (at www.digitalocean.com)

Any of you had that problem?

Couple of ppl has this high cpu usage problem on stackoverflow, but no answer.

haven't seen anything useful in logstash logs

fdrolet: What do you mean you cannot kill it?

pkill -9 java

does nothing

service logstash stop does not work

status says it's not running

...

so what's using your proc?

9402 logstash 39 19 2072416 121896 13176 S 185.1 2.9 0:07.46 java

110 logstash 39 19 2992448 225212 16056 S 0.3 5.4 0:25.38 java

kill -9 9402 ?

kill -9 110

java processes keeps respawning

i tried that

oh

What's pstree say?

I am pretty puzzled

hum

any useful flags? i'm not used to pstree

pstree logstash

see what's starting respawning those

# pstree logstash

java---53*[{java}]

java---18*[{java}]

...

pstree -a

YOu dont see any like fork things

# pstree -a logstash

java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC-XX:+U

`-52*[{java}]

java -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC-XX:+U

`-18*[{java}]

does killall -9 java do anything for you?

fdrolet: it's probably just that logstash-web is running

nope

what's the *full* args to that process that won't stop?

yeah do a ps -awwwwwux and see what the full args are

logstash 11717 19.7 5.5 2992948 231764 ? SNsl 15:24 0:17 /usr/bin/java - java.io.tmpdir=/var/lib/logstash -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/co

nf.d -l /var/log/logstash/logstash.log

i have 2 of those

what is logstash-web?

doesn't seem to be running. I put an "exit" in each init scripts (elasticsearch, logstash, logstash-web) so they don't do anything but logstash still runs after a reboot

seems to be started from elsewhere

and dies but respawns

on debian systems is it /etc/init.d ?

or rc.d

look at hte scripts of your runlevel

init.d

grep them for java

only those 3 I mentioned

well

remove all threee scripts from the runlevel directory

reboot

just did that, even shut the vm completely. still there 0_o

found the culprits. there where executable scripts logstash and logstash-web in /etc/init

in addition to /etc/init.d

first time I see that

they are called logstash.conf and logstash-web.conf, so I'm not sure if they are supposed to be executable

OK so after moving away /etc/init/logstash.conf and logstash-web.conf, logstash doesnt run after a reboot, so that's one mystery solved (for me anyway). Now I have to understand why the processes keeps respawning and using lots of cpu time

must be something wrong with my config, but as I followed digitalocean's guide and don't see any error in the logs, I don't really know where to go. Thanks for your help on that first part anyway :)

i really want to use this elk stack because it seems awesome. I did some filters, setup some forwarders and it's great. It still works, I see things in Kibana, but I know this high cpu usage isn't right. Don't want it to bite me after I take the time to setup filters and forwarders for all kinds of services and servers

sorry I'm probably rambling

fdrolet: just catching up (was at lunch) but it seems you're an ubuntu user. Upstart is the init system Ubuntu uses, and it looks at /etc/init/*.conf

logstash ships /etc/init.d/ scripts for things as well, and this is where it all goes wrong for ubuntu folks

Typically, ubuntu uses a wrapper script as a symlink in /etc/init.d, so that "service foo stop", or /etc/init.d/foo stop still cue upstart

but logstash is shipping a real init script there, since it uses that on non-upstart platforms

torrancew: you are right, this is on ubuntu, but I had the same problem on debian

fdrolet: which debian version?

7.7

I have not seen that behavior, and am a deb7 user

Any advices for list vs channel with Logstash / Redis?

if you can share your findings in real-time, happy to look into it with you

torrancew: great, thanks for these init pointers, I am still confused with it sometimes, even after 10+ years of linux admin ;)

:) no worries

so are you saying that there are 2 different init scripts (one in /etc/init and one in /etc/init.d) and that breaks ubuntu upstart?

are these 2 compatible?

not quite

basically, ubuntu is using upstart