#logstash

My overall symptoms are that I just setup logstash

and it works if I keep restarting the logstash server

the stdout logs are written 2 - but it just stops

some relevant errors

{:timestamp=>"2015-05-29T13:33:46.065000-0700", :message=>"Got error to send bulk of actions to elasticsearch server at localhost : blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];", :level=>:error}

and..

Failed to flush outgoing items", :outgoing_count=>65

the 2nd one I have found hits on the internet - and they talk about mismatches between Elastic Search and LogStash

but the versions that I am using come from article on digital ocean - so I have to assume it was tested

said artical: https://www.digitalocean.com/community/tutorial...

<http://tinyurl.com/od69d85>; (at www.digitalocean.com)

t4nk524: even if things are setup properly, they can still go off the rails :) "no master" sounds like your cluster is unhappy ?

everything is running on localhost

lumberjack sending from systems to a logstash instance and one ES node all on one system

Is the flow through the filter section described anywhere? I haven't yet found anything explaining when filter processing ends for an individual event. For example, can one event flow through two different "if" statements if it matches both? Is this documented anywhere?

pcrook: I don't know if it's documented, but filter sections are a set of instructions. The event goes through _everything_

it's only skipped if you use a conditional and that condition is not met

you can nest 'if' blocks, or have a million 'if' blocks, whatever you want

rastro: :'(

whack: at least i have a goal now :)

whack, so an event only falls out at the closing curly brace. That's what I needed. Thanks.

pcrook: drop{} will also kill it before the end. ruby code, too.

pcrook: it's supposed to work similarly to other programing languages, where things are executed in order and branches are taken if the conditions are met

(where branches are if, else if, and else)

I have an error log which prints out java error output. Some of the messages are multiline stack traces, but others are single-line debug outputs. Is the correct way to filter these out with 2 different filter blocks?

rastro, are you saying that certain Ruby statements will terminate further filtration or that any use of the ruby filter jumps out?

pcrook: the drop{} filter will stop filters and output for that event. You can do the same in ruby{}.

rastro and whack - thanks again

<3

when logstash is going crazy (like 100% cpu and nothing arriving on elasticsearch) is there a simpler way to find out what is going on than debugging java like this http://www.semicomplete.com/blog/geekery/debugg... ?

<http://tinyurl.com/3g8l64s>; (at www.semicomplete.com)

kit_: there's no much visibility into LS (yet). Do you have a stdout{} output?

you can add --debug to the parameters when starting

also look at the log files

kit_: did you try the `top -Hp logstash_pid` as recommended in that url?

kit_: because that might help start your path to debugging

pauldev-work, depending on how those 2 types of logs are printed, you might be able to handle that with just one multiline

hey, how do you use output pipe plugin to pass the output to rotatelogs

do you mean depending on if it's printed by the same process or am I misunderstanding?

<pauldev-work> do you mean depending on if it's printed by the same process or am I misunderstanding? --- if log format of the first line in both cases is the same

RobertDupont: yea it is the same. First line is basically date, log-level, exception name : debugging output, then line terminates. If there's a second line it's initiated by stack trace

pauldev-work: nvm, i got it :)

oh, sorry echelon didn't know you were talking to me

pauldev-work: oh, i thought you were responding to me..

haha, nawp, responding to RobertDupont

:]

anyway, i did.. output { pipe { command => "/usr/sbin/rotatelogs /path/to/log 20M" } }

that is the equivalent of max_size

which isn't implemented

for output to file

Y'all got any tickets you want me to look at?

whack: I'll do the top -Hp next time (I've restarted the service some hours ago), I got nothing on stdout nor in the logs

yup

whack: yes, but it's on my private trello board at work, and it's unrelated to logstash :(

<http://tinyurl.com/pxedqsk>; (at github.com)

oh sorry, I answered too fast, yes I tried everything in this url, anyway LS was hung on connection limit by LSForwarders, I'll try to do the top faster no next problem

whack: https://github.com/logstash-plugins/logstash-fi...

<http://tinyurl.com/oocz96y>; (at github.com)

s/no/on

jkitchen: I think it's best I don't have access to your trello.

jkitchen: I might ... move things.

is there a best practice for capturing a stack trace or debug output when logstash is crashing like in issue #103?

m0nky: running logstash with --debug will show a stack trace on exceptions

is it best to just pipe that out to a text file?

whack: yes, move them to done, after doing them, pls advice, thx

thing will be massive, i assume

rastro: date#23 should be easy fix

whack: yes, a bug with kv

whack: looks like i've only been +1'ing kibana tickets! Lucky you :)

whack: https://github.com/logstash-plugins/logstash-fi...

<http://tinyurl.com/n98ekso>; (at github.com)

rastro: hah, I ignore +1's on tickets :p

whack: yeah, but it helps me find them when you ask for tickets :)

RobertDupont: noted, kv#12 should be an easy fix; I'll try to write a test to reproduce

m0nky: you can use the --log flag to have logstash log to a file

awesome

whack: thanks, i'll give it a whirl

<whack> RobertDupont: noted, kv#12 should be an easy fix; I'll try to write a test to reproduce --- you can take what's in the ticket

s/ticket/issue

yep! will do

RobertDupont: I can't reproduce it :(

weird

I updated the ticket with what I tested

I'll give it a try again

rastro: not sure I'll be able to do date filter #23 today, right now the timezone handling is done at plugin startup time, not during parsing

How do I get a "remaining text" match when using multiline? I want a "logmessage" field that's everything past the fields I extracted, but .* matches only to the next newline. Is this where a greedy matcher will help?

pcrook: put (?m) at the beginning of your pattern

oOo

you mean "(?m).*" ?

show me your config?

just the 'match' line

pcrook: what version of logstash are you using?

whack: I

whack: I'll see Monday with the IT guy to make the firewall send that log again

whack, 1.4.2

(he's not here today)

pcrook: it defaults to matching across newlines in 1.5.0

RobertDupont: ok cool :)

RobertDupont: sorry I couldn't reproduce :(

"message" => "%{CATALINALOG}"

CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}

JAVALOGMESSAGE (.*)

pcrook: on logstash 1.4.2, do: "message" => "(?m)%{CATALINALOG}"

<whack> RobertDupont: sorry I couldn't reproduce :( --- I tested and got the same results as you

whack, OK, but what is that prefix?

whack, and why put it on the entire line match rather than just on JAVALOGMESSAGE?

whack, works like a charm too

pcrook: you can put it on the JAVALOGMESSAGE pattern if you wish

but it's different syntax then, I think

I think you'll need to do (?m:.*) Instead of just .*

whack: if you're looking for another simple bug, I remember something with WMI plugin missing some code (happened during the move to individual repositories) for remote WMI. Or maybe it was eventlog