#logstash

_catch_: http://www8.hp.com/us/en/software-solutions/sof... <-- oh look, debian/ubuntu isn't supported. boom! logstash wins \o/ (imho at least)

<http://goo.gl/fmUfQ>; (at www8.hp.com)

We can't even use debian/ubuntu. It's pure madness.

should logstash hold open log files that haven't been written to in a while? it's holding open logs from days ago...

hopefully you wont find anyone here seriously trying to prove some other software is shitty. usefullness depends on the use case, and different people have different criteria

in kibana, by default, it shows up 'Time' and '@message', is it possible for it to also show 'syslog_hostname' by default?

jameshyde: edit the KibanaConfigrb

+.

there's a default fields setting in theere

whack: i'm running redis on a VM with 3cores and 6gb ram, with redis fsync and all persistence disabled.. no packets where dropped over the network

mortini_: thanks, it works!

cool

whack: is there any extra setting/tunning i should apply ?

I am not sure why kibana works for chrome browser, but not for IE8 browser here.

hrm logstash isnt closing files because im using discovery on a glob, and i'm putting logs in the same dir, just datestamped. is there a way i can force the file input to match a regex for the filename?

jameshyde: requires html5 support

i don't know ie8 has it all

probably need 9 or something

ok, thanks!

back

jameshyde: this just happened on Friday... https://github.com/monitorama/hackathon/issues/10

Title: Improve IE8/9 support in Kibana · Issue #10 · monitorama/hackathon · GitHub (at github.com)

semiosis: nice!

attempting to get tomcat brought into our logstash config - what a wonderful mess of weird patterns in the catalina.out. One of the log entries is "date class method\nlevel: text". Should be easy enough, match the first line and automatically include the next.

that being said, multiline for me seems to require pattern - here's what i'm attempting to use: http://pastie.org/7272627

Title: #7272627 - Pastie (at pastie.org)

`log4j

semiosis: I do not know about 'log4j', but I do know about these similar topics: 'log4j json'

jlambert121: have you looked at ,,(log4j json) ?

jlambert121: https://github.com/lusis/log4j-jsonevent-layout

on the todo list, not an option today - internal stuff

yep thats pretty common

if i don't supply pattern for multiline i get "Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (TypeError) can't clone NilClass"

the docs has it listed as optional though - trying to just match a tag

sad that better solutions have to go through the game, isn't it?

are you using flatjar or monolithic?

monolithic

sounds like a bug... if pattern isn't required and there's no default, then it's nil which causes an exception

:(

i think pattern should be required, dont you?

Can you grok multiple filters out of a single log? For debugging purposes, can you log grok failures so you can fix or add more grok filters?

right now my answer is now, but maybe i don't know "the better way". if you look at the sample messages on my paste do you have a better idea for how to include the next line with a grok match?

s/now/no

I should say….grok multiple patterns...

jlambert121: what => "previous" pattern => "^[A-Z]+:\s+"

Evening.

afternoon

semiosis: sure - was just trying to avoid the bit of overhead of processing a pattern - i'll give that a shot though

semiosis: should i enter a bug for multiline on jira?

techminer: you can give grok an array of patterns, it will try them left to right until one matches, see ,,(modsecurity) for example

techminer: mod_security crs log parser: https://gist.github.com/1346387

jlambert121: couldn't hurt to file a bug

So I am taking a stab at using Logstash as an alternative to services like Splunk because Logstash doesn't cost my first born. Just launching the Logstash monolithic JAR file with ES embedded server, it takes a minute or two to load up with a fresh install - how come? IS there anyway of me seeing what's going on?

semiosis: thanks for your help - i'll file a bug on this and test this pattern when logstash restarts again :)

never heard of anyone using mutliline without pattern

if there's a legit use case for that i'd like to know what it is :)

yw, hope it works

The VM (backed by KVM) only has two vCPUs I guess.

semiosis: i assumed adding a tag at a previous step than just matching on it later would be the fastest processing time

premature optimization at best, not going to work at all at worst

make it work, make it right, make it fast

in that order

fair enough

semiosis: thanks….so create a file in my patterns dir with all the matches I want to make from a log….and include that patterns_dir in the filter….

in my opinion anyway

trying to think too much about it. i don't diagree

Is it possible to run Kibana on https only? Or some other recommended way of securing it?

see ,,(kibana passenger) and run it inside apache

semiosis: Two gists to help run kibana on passenger... in the site root? https://gist.github.com/3794465 -- or in a subdirectory? https://gist.github.com/4658921

it appears logstashbot has a lot of good references. is there a listing of it's sources and tags one can browse?

you can "/msg logstashbot factoids search #logstash *" to get a list of factoids, then issue each one as a separete message to see the values

thanks

`factoids

semiosis: Error: "factoids" is not a valid command.

i'll give that a shot later tonight - wondering how many questions it can answer for me

maybe :)

`learn factoids as you can "/msg logstashbot factoids search #logstash *" to get a list of factoids, then issue each one as a separete message to see the values

semiosis: OK.

a meta factoid

jlambert121: there are not many factoids yet, the bot is still relatively new

New news from newjiraissues: Justin Lambert created LOGSTASH-980 - Multiline documentation incorrect <https://logstash.jira.com/browse/LOGSTASH-980>;

semiosis: it appears to know more than i do though - i'll take what it knows to help start

<http://goo.gl/PlNX0>; (at dl.dropbox.com)

That legend format scales a bit better

rashidkp1: aren't you supposed to be vacationing?

semiosis: Am I?

was hoping to see a graph of wave heights

in June :-)

ohhh

I need to pick a better color scheme

rashidkpc: HAI