#logstash

they won't reconnect

that's the thing - logstash isn't accepting new connections

I have a feeling that because its incoming connections are stuck in CLOSE_WAIT, nothing happens

well there is probably a connection limit

or could be rather

hello,

i have set a grok pattern, which works in the online debugger, which parse all the data I expect it to parse, but still tag my log as `_grokfailure`

how can i investigate ?

calve: tab/space issues?

definitely not

I mean, i exclusively use spaces, and I am not aware of any limitations

grok is not yaml, isn't ?

my config is some syslog client read logs from files on the host, send it to logstash on a tcp input

calve: in your own log?

so I've found apps like to mess with my life by using tabs and or spaces.

I ended up writing some mutate stuff to just kill them with fire.

it is a standard apache application

so the gotach is copy the line out of the terminal paste into grokedebugger and it's now spaces and everything in the pattern works

so that's why I mention it.

I understand

kyrill: ok, that's more data to work wiht (sorry, was on a call)

the thing is that logstash actually parse correctly the log

kyrill: how many clients (LSF nodes) do you have?

i can see all my custom fields in kibana, and none of them appears to be missing

calve: you only have one grok{}? Are you using the syslog input{} ?

calve: ^^^ rastro is asking all the right questions

rastro: i have multiple grok in my configuration, the other one works as expected

and no, i am not using syslog input

calve: are you using tag_on_failure?

rastro: no (can't find that string in my conf)

calve: it's a feature of grok{}: http://logstash.net/docs/1.4.2/filters/grok#tag...

<http://tinyurl.com/ko886fy>; (at logstash.net)

maybe the lack of GREEDYDATA is making it fail ?

calve: you should set a unique one for each grok.

calve: more likely your event is passing through more groks than you expect

and a /different/ one is tagging it _grokparsefail

calve: you can match part of the input field; it doesn't have to be exhaustive.

i will try a different tag_on_failure for each grok, and see the one that is failing

waouh

i have no failure anymore

nice

thank you very much rastro and torrancew

calve: well, just adding tag_on_failure wouldn't fix the GPF...

i cant read one of the tag i specified in kibana

getting somewhere now - looks like a few hosts are doing their level best to flatten the relay

is there a setting in logstash-forwarder to limit the batch size?

kyrill: a cli arg

-spool-size IIRC

yeah found it

ian_mac, I think I've found the cause

it looks like the latest version of logstash-forwarder defaults to a batch size of 1024. Previously it was 100 (for me)

120 nodes, all relaying to a single endpoint in batches of 1024 events flattened it.

that's my prevailing theory