#logstash

anyone know of a decent "json log viewer"?

I will go blind looking at this json output from logstash

I'm on a Mac

New news from newjiraissues: Martin Cleaver created LOGSTASH-1193 - Is there a way to have verbose logging for my plugin but quiet logging for the rest of logstash? <https://logstash.jira.com/browse/LOGSTASH-1193>;

I suppose I should be looking at my logstash logs through kibana

Anyone know how to upgrade the metadata correctly upgrading from 2.2.5 to 2.4.5 using config metadata since 1.x?

er, wops, wrong channel :D

lol :)

Anyone ever used logstash to parse OpenLDAPs logs? Since it's multiline I'm thinking grok and `break_on_match => false`, anyone got another better idea?

simmel: Multiline?

simmel: http://logstash.net/docs/1.1.13/filters/multiline

Title: logstash - open source log management (at logstash.net)

basti: Ooh, looks nice!

hu, just send a few thousand lines to logstash for testing. Redis queue is really fast growing. I guess indexer/elasticsearch is too slow to handle these queries. I already did all tipps from: http://jablonskis.org/2013/elasticsearch-and-lo... . Any further ideas. How many lines/s are possible

Title: ElasticSearch and Logstash Tuning | vaidas jablonskis (at jablonskis.org)

basti: IIRC electrical has a setup that can handle 250k entries per second?

basti: I managed to reach 1500 events per second, though it was a somewhat convoluted test environment

(that could be way off)

electrical: Another hot tip for me? ;)

With what kind of hardware you did it and which tuning config for indexer/elasticsearch?

basti: no tuning, mostly a vmware machine with 2 cores and 4 or 6 gigs of RAM

hi basti: the machine i tested on was: 32 core, 128GB mem, 2 x SAS disks ( Raid 1 )

Though from that experience it _really_ pays to separate logstash and ES

got about 30k/sec but should be able to do much more

yeah 1.5k I currenty have. atm 6-7k are send to it

each logstash output can do about 3-5k/sec so you will need more logstash instances for pushing to ES.

basti: advantage of logstash is that you can have multiple instances processing the logs from the queue

ah okay, simply configure them to use same redis queue with same config?

Yup

yeah indeed

Trying it . .

(currenty vm with 16 cores and 10GB ram)

viq: the 250k/s setup is mainly theory at the moment. but the idea i have is pretty sound

the setup though requires allot of machines for different functions.

When using multiline is it possible to save a reference from the "old" message to the new one?

simmel: What do you mean by "reference". Do you mean the "what" setting?

basti: Not really, the OpenLDAPs logs looks like this http://pastie.org/private/sqa7vo2vypvclxxg6trebq and I want to find the lines with "MOD" but I'm also interested in if the RESULT is "err=0". So I thought I could: 1, Find the "MOD dn="-line and store that as a uid tag, store "MOD

Title: Private Paste - Pastie (at pastie.org)

attrs=" as a attributes tag (preferably as an array) but only if "conn=from the previous MODs op=also from the previous MODs.* err=0" matches.

I'm using logstash with udp input coming from a distant syslog-ng. I very frequently get grokparsefailures, and it seems the syslog message is prefixed with a diamond character, e.g.: '<29>Jul 8 14:42:25 hostname/hostname ...'

any idea how this could be solved?

faxm0dem: pattern => \<%{NUMBER}\>, rest of your grok . . .

Or strip it via syslog and a template

right, but I was wondering where this could come from

Syslog default template

electrical: Currently running 24cores@32GB with 4 instances and it is still much to slow. Any further hint for me

basti: syslog default template?

faxm0dem: simply add to your syslogconfig template("$MSG") , than it is only sending the message without syslog timestamp and faciltiy, hostname

faxm0dem: There is a default template inside syslog-ng, which applies when you are not setting an own

basti: right, I wasn't even looking at syslog-ng config. thanks!

electrical: damn, io seems to be bottle neck

basti: what kind of storage is connected to it?

I am just migrating to ssd's.

SAS had been there

okay.

even with sas i could do allot ( 30k/sec )

When testing a config, is there a shortcut to make logstash (agent) forget that it has ever parsed a file input already?

currently i have permantly 100% utilisation

simmel: you will need to configure it to start at the begining of the file, and manually remove the .sincedb file

basti: oops okay

basti: if it's disk io that limits you, see if you could enable compression somewhere along the way

Either filesystem compression, or make ES compress the data

perhaps change the flush time?

hi I'm running log stash 1.1.12 and it seems like the client has a memory leak of some kind. Over a period of a week or so it fills the memory dedicated to the jvm memory (the default 384mb) and starts to swap at a steadily increasing level

once restarted the cycle happens again

viq: nice idea. It is already enabled for version 0.90 in ES. I have 0.90.2

I'm only shipping logs to redis, and running syslog through log stash too.

willejs: did you limit the memory that logstash is allowed to used ? ( -xmx i believe )

electrical: Sweet, that worked. Thanks!

simmel: np :-)

electrical: yes, to the default ~384mb

willejs: if the box is swapping that implies that you haven't got 384mb+ spare -- the JVM can't exceed that Xmx setting. A mem leak would show up as back-back Full GC and logstash would lock up

CasperGasper: cheers you may be right i may not have xms set

Xmx -- Xms is initial heap

so i know i have xmx set, but if i set booth to the same value it will ensure it won't swap

providing i have enough memory ;)

it really just minimizes pauses for the heap expanding

electrical: How high should be Xms in my setup? currenty its at 1MB (still much to sloow) 3GB XMX and running 3 instances of logstash indexer

it's probably best to set Xms and Xmx to the same value

That means, that every java thread has as much ram as the hole process is allowed to use. Or am I wrong?

basti: no, it just means the initial heap size is the same as the max heap size --

saves time expanding it later on demand

CasperGasper: ah okay, i am trying this ^^

Ouh, talking about xss ^^

xmx and xms should have of course same size

basti: as it turns out, the grok failures were due to the hostname field containing a slash: 'hostname/hostname'

basti: not sure as to why this should happen

faxm0dem: So wheres the Problem?

hi, I run logstash with java -Xmx1024m -jar logstash-1.1.13-flatjar.jar agent -f logstash.conf -- web --backend elasticsearch://156.123.17.18. The crashes with following error when I access with the logstash web client:java -Xmx1024m -jar logstash-1.1.13-flatjar.jar agent -f logstash.conf -- web --backend elasticsearch://156.123.17.18. Any thoughts on how I may resolve this?

basti: the problem is the pattern I had to replace %{SYSLOGHOST:syslog_hostname} with %{WORD:syslog_hostname}

I guss "/" is not in class "WORD"

which works fine but I don't understand why I get these weird hostnames in the hostname part of the message

okay :-)

Some syslog magic

I am getting now weird error message: "Failed to index an event, will retry {:exception=>org.elasticsearch.action.UnavailableShardsException: [logstash-2013.07.08][14] [2] shardIt, [0] active : Timeout waiting for [1m]," Which timeout is meant`

stupid question: do all plugins on the logstash doc page ship with logstash, or do I need to track them down somewhere?

axellj: those on the page are built in

grasshopa: I think very few people use the builtin web client, everyone seems to use kibana instead. I'd recommend trying that.

basti: my uneducated guess is that ES took over a minute trying to write a shard?

grasshopa: also it seems you're using integrated elasticsearch, it is really much much better to run a real one

viq: Well, but why should there be a Problem and how can i debug it.

viq: thanks!