#logstash

gotcha

come to antwerpen

we got beer :P

rashidkpc, if ur smart you go in early, find a nice spot and stay there :p usually near a bar :p

i think the amstelveld is the winner

central, not just techno djs

hehe yeah

hi

im off to home. take care all

Some of my log messages have an attribute called 'level' and some doesn't. How do I use mutate(?) filter to add a default level for those messages which doesn't have it by default?

hi all

howdy Electrical

how you doing devOpsEv?

not bad, hanging in there

hehe okay :-)

hey all

so I managed to get grok working over the weekend, and I'm pumped. I have a question though. All of the fields that it is creating are prefaced by @fields. in the search. Do I need to use mutate to drop that prefix?

sudoSamurai: no, and you don't need to use the @fields in the search in Kibana

oh, sweet. Check that out

pretty much for everything, ignore that that "@fields" bit is there

I believe it's going away in Logstash anyway

very cool

I'm getting closer to something workable

I'm getting an Output thread exception {:plugin=><LogStash::Outputs::Gelf .... NoMethodError: undefined method `downcase' for nil:NilClass

I feel that it's because I'm trying to %{myvariable} in some of the parameters, but the message doesn't tell me which parameter nor in which message :(

Garo_: paste config and log output somewhere so we can take a look

Garo_: what are you downcasing ?

%{myvariable} ?

Could be that the variable you use in the gelf output can be empty? when its empty ( not existing ) it throws that kind of error

Spredzy: Logstash / Gelf output don't allow to directly downcase in the config, I believe that it's in the code

devOpsEv: Actually I was trying to guess where the downcase could be called from since it says nil. I - yet - never played with Gelf output

devOpsEv, Spredzy: https://gist.github.com/garo/7c4db57ecb299094b4ff

Title: gist:7c4db57ecb299094b4ff (at gist.github.com)

Well it's trying to downcase in the gelf output

yep

your second log message doesn't have a level

yes. One of the first two grok filters should match, so there should be "level" in there.

if the level is missing, then the level => "%{level}" rule in gelf output will trigger the exception

yep

but the exception doesn't say if it was "level" which was missing, or if "where" was missing etc

well either way you need some more specific filtering/outputting to avoid the exception

Garo_: that's because the debugging isn't that extended :-)

so it's kinda hard to debug. obiviously there's some log message which will trigger this, but I don't have yet any good ideas how to find it

New news from newjiraissues: Dick Davies created LOGSTASH-940 - 1.1.10-dev amqp/rabbitmq output causes high CPU usage <https://logstash.jira.com/browse/LOGSTASH-940>;

I'll try to turn the stdout output on to try to catch the message which caused the exception

Garo_: are you using -vv?

no

use cmd line flag -vv when launching logstash, make sure oyu're logging to file

that will give you the most verbose debug output

ok. thanks. I'll try that

it may not give the exact debug you're looking for but it will give you the parameters being sent by gelf output

which could tell you which one is blank for each message

Garo_: did you make sure you had no _grokparsefailure ?

phrawzty: ping

Spredzy: he has tag_on_failure = false

Electrical: pong

devOpsEv: yeah, I've currently designed the filters so that one of the first two grok filters need to match

I see what's happening though

Sorry I just verified what tag_on_failure was actually doing, my bad

when something matches the second grok pattern

level and where are both going to be blank

so those messages will always throw an exception on that output

because of the add_field => ??

as will ip

yes

adding the field but not populating it with anything before the output

but the add_field will set level, where and ip fields?

it will add the fields but not put values in them

I'm pretty sure it will set level => "debug", where => "N/A" and ip => "N/A"

at least that's what I'm getting into graylog2

oh duh

yeah sorry, forgot that was a hash

no problem =)

use ' instead of " around N/A

literals and all that

guess it shouldn't really matter

Im lost on one thing, was your statement correct devOpsEv ? On the fact that it will create level with blank value ? Was that the error ?

no

no, add_field is a hash so with one thing like add_field => ["some_field", "some_value"] some_value will be the value of some_field

I just wasn't thinking

ok

hey all, in the "pattern =>" area, is there a way to make one or more of the fields optional?

Garo_: debugging would be easier if you output to something like ES and could use Kibana to verify your fields are populating correctly

sudoSamurai, "pattern => " for what, grok?

yeah, sorry

in grok

no, it's regex so you can do whatever field validation you want

but you can't make a field optional

I'm looking at some logs, and if there is an "ok" status, it ends there, but if it fails, it is followed by the error message. Right now the only messages that are being split into fields are the error lines because the OK lines are missing a field

you can use | as an OR statement

devOpsEv: thanks for the idea. I might try that out

. will match a space

I just first tried out graylog2 as many suggested it as a better alternative than the logstash own frontend

so you could do (.+ | someotherpattern)

Garo_: yeah, but the logstash web interface sucks. Kibana and ES are not Logstash

this is in the grok pattern statement?

sudoSamurai: .+ will match blank spaces

sudoSamurai: and the | says either

Garo_ you did checkout kibana? :-)