#logstash

nginx input plugin? huh?

you mean an official nginx sample?

that is a strange request

fatdragon: https://github.com/elastic/logstash/commit/d6a6...

<http://tinyurl.com/zv7ltnc>; (at github.com)

berglh: oh. so it was fixed or should I put the fix? What version was the fix? how do I check my version?

I am on 2.2.2

it was fixed a while ago as far as i can see

what's the warning you're getting in your debug log?

debug log is going, I don't see any warnning right now. but after few days.. filebeat can't send anything to LS and LS just starts to stall and refuses any new connections..and I have to restart LS routinely every few days.. I have tried..increasing memory, Xmx, heapsize, entire system memory, turned off tcp6 calls to ES.. now waiting for another crash tomorrow or next day to get a java dump..

some sort of memory leak or bottleneck..somewhere.. I do see lots of TIME_WAITS from LS to ES, not sure if that's any problem..I have 150 timewait connections then it goes down to 120 and back to 140 so it is flushing I suppose

it soudns like mayeb your logstash can't write out to elasticsearch as fast as you're receiving messages

i that case i'd expect it to use a lot more head

heap

are you using a messaging queue?

like filebeat > logstash > redis/rabbitmq/kafak < logstash > elasticseach?

in this case if the logstash can't write to elasticsearch fast enough, the message queue will take the load

I am new to ELK , just running FB > LS > ES > Kibanan, I suppose

yep, that sounds like your problem then

maybe

what filters are you using in your logstash config?

is your ES isntance running out of heap?

maybe that would explain slowness in ES

all three LS, ES, Kibana is on one server with 8GB memory

I have 3 GB allocated for ES and 3GB for LS, not sure if they are being set correctly ..using LS_HEAP stuff in init scripts

its just a standalone ES config

ES continues to run when LS bums out?

I have beat input, syslog filter, elastic search output config for LS

ok, doesnt' sound too complex

yea.. ES is fine

very simple setup, 7 nodes..low traffic with auth and syslog and stuff

just trying to get off the ground and stablize it so I can add more nodes

do I need to install redis?

or tune ES or LS?

what I can do is get a java dump when it crashes, hope I am awake and am able to catch it, before LS java process dies..

curl http://localhost:9200/_nodes/stats/jvm?pretty | grep heap_used_percent

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 2270 100 2270 0 0 358k 0 --:--:-- --:--:-- --:--:-- 369k

"heap_used_percent" : 69,

that seems kind of healthy

when it's failing, check out that heap usage

on elasticsearch

will do..thank you

there are a bunch of other errors you can see in the elasticserach log

that indicate resource busy

so like hard disk contention

the logstash problem is strange though

I would say it should stall/crash around wed from experience, so I'll keep an eye..

let me check my LS instance usage now

protip: curl -s to get rid of that stats rubbish when you're piping it elsewhere

yeah, curl -sS

:D

ALL THE THINGS