#logstash

okay.

then you could use it for doing that indeed.

Hello, I have a question about the new release of Kibana milestone 3

Renaming 'logstash.json' to 'default.json' doesn't work.

perfect, thought so. not really looked into what kibana can do for me, just wanted to make sure i could use the tech

gmast, fire away ( not sure if i can help you but will try )

:D

I keep getting the introduction page.

gmast, you put those under the 'dashboards' dir /

I think a found a bug.. Firstly, I renamed 'logstash.json' to 'default.json'

Then I refreshed. That didnt work.

Secondly I visited the url ... url/file/default.json

Then it worked

I worked only after I visited /kibana-master/index.html#/dashboard/file/default.json

Strange.

Thanks anyway :D

hmm okay. bit strange. not sure if that's expected behaviour. perhaps ask rashidkpc when he's online

Yea. I will.

electrical : my question is match both pattern1 and pattern2

vks, don't think that's possible.

electrical, ok

euhm sorry. yo ucan do match => [ "@message", "%{pattern1} %{pattern2}" ]

you can add multiple small patterns into 1 match.

http://logstash.net/docs/1.1.13/filters/grok <-- like in the example parts

Title: logstash - open source log management (at logstash.net)

seems i'm not completely awake yet :p

for a new aplication (no log format decided yet) for which I want to send the logs to logstash, how should I format them? json or something the kv filter can parse

sagarv, if you want to do no parsing ( and its possible ) you could put it into json_event format.

sagarv, http://cookbook.logstash.net/recipes/apache-jso... <-- example on how to do it ( this is for apache )

Title: make apache log in json - logstash cookbook (at cookbook.logstash.net)

electrical: yes, was just looking at that. looks like that would make most sense. thanks

np :-)

hmm, noticed that kibana will require es 0.90 soon.

hope not log stash requires 0.2x

eper: elastisearch_http

logstash 1.2.x will be working with ES 0.9x natively

yeah but _http requires a IP/host

its not very good for a cluster

would need to put load balancer in place to make it use several nodes or use another node if specified is down/etc, ES client joins and knows about everything already

ok then you'll have to wait for 1.2

yeah but 1.2 will be the way forward

any time from on logstash 1.2?

there is no ETA on it yet.

if i have a grok pattern that matches my input perfectly on the grok testing herokuapp, but doesn't work in actual logstash, what the hell's going on?

Help!

(first time irc user :-)

fire away with your question(s) and some one will try to help you :-)

thx

ok. cencerning logstah. did the tutorial http://logstash.net/docs/1.1.13/tutorials/getti... with the embedded elastic search server

Title: logstash - open source log management (at logstash.net)

when tryin to start logstah got "ava.io.StreamCorruptedException: invalid internal transport message format"

what're people loving for shippers nowadays?

seems like the embedded ES server isn't started. stacktrace on screen didn't help me.

SKAR_, can you pastebin your config and startup commando?

merci

conf: input { stdin { type => "stdin-type"}} output { stdout { debug => true debug_format => "json"}}

start command: java -jar logstash-1.1.13-flatjar.jar agent -f index.conf

Maior, beaver is widely used. ( python ) or lumberjack ( C )

the more simple sample without embedded ES worked fine.

SKAR_, you only have the stdout config so no elasticsearch output ( and thus no embedded one )

sorry.

pasted the wrong config

hehe okay

please use pastebin.com or pastie for configs. saves allot of text on IRC

finally I know pastebin ...

Title: Here's the config file named index.conf: input { stdin { type => "stdin-type" - Pastebin.com (at pastebin.com)

weird you are getting that error. which Java version are you running?

electrical: cheers

Maior: one thing I've seen with beaver is quite a high cpu and memory load

still need to look into why

sorry. I'm too stupid to paste console stuff to bastebin...

will try to paste the full sysout. just a moment.

here we go: http://pastebin.com/iB6hHsR6

Title: Java version: java -version java version "1.6.0_26" Java(TM) SE Runtime Env - Pastebin.com (at pastebin.com)

do people run a lumberjack instance per source?

or One True Machine-Level Lumberjack?

SKAR_, okay. 1.6 is supported. so that should be fine as well.

not sure what's going on SKAR_

Maior, not sure tbh. never used lumberjack

did use beaver ( because of the redis output )

Anyone ever used rsyslog with the omprog module and a shipper like beaver or lumberjack?

Maior: when I played with is one lumberjack per host

@electrial: thx

I think I skip the tutorial and try to do a proper setup

without the embedded es

okay :-)

if you have any questions, im usually here

Maior: moved to beaver for amqp support (rabbitmq)

trying to work out best way to handle app logs

really quite liked the idea of having my app packages drop config in `/etc/logstash.d` or similar

we use puppet to deploy beaver (beaver support conf.d config loading style) then beaver -> amqp -> logstash. Nice way is that this way I can add per app tags in the beaver config

sagarv: ah it does? brilliant

sagarv, using my puppet modules? :p

lumberjack docs are a little light

*feel a little light

sagarv: Have you tried using redis as the middle cache? I'm curious on rabbitmq vs. redis for middle cache. (We use rabbitmq now for ESB-stuff)

electrical: I wanted too but using foreman so I can't use defines

sagarv, ahh okay. to bad

sagarv: Also, are you using HA on the queue? Persistance and durability? Any other setting that you've find that works wonders when using rabbitmq for logs?

From September on i have much more time to spend on the puppet modules. might spend some time to see if i can get it to work with foreman :-)

I did use your module as a base though. greatly documented and I like the structure of your modules :)

Thank you :-)

sagarv: And are you mirroring your queues?

electrical: plan to play around with it as well. found a way (not pretty) to kinda hack around the lack of defines

sagarv, okay :-) lets take a look at it somewhere in September. see if we can find out a way.

simmel: at the moment a pretty basic setup. two rabbitmq nodes in a cluster, pubs and subs connects on a RR basis. I have however not done any of the fancy HA stuff, like queue monitoring

sorrym queue mirroring.

Not sure if I want to go through the hassle of that to prevent loosing any log messages. I'm fine with loosing a few

electrical: sounds like a plan

sagarv: Ah, ok. Are they disc or ram replicas?

simmel: disc

sagarv: It's always nice to be able to trust your logs. We have the trust problem now so we want to get away from that = /

Cool, thanks.

sagarv, i have a few idea's that i can think of but will need to look at foreman as well. haven't played with it yet :p

simmel: yes you don't want trust issues :) you bring up valid points though and remind me having to look at more ha stuff for rabbitmq.

sagarv: = ) rsyslog with relp is supposed to help too but I can't really understand if it buffers when it can't transfer the logs (but lumberjack does that AFAIK)

sagarv: are you using logstash master or 1.11.x?

electrical: for simple you can use a smart variable in foreman (I use an array) and then call a define function with that

electrical: http://pastebin.com/xiiL2UW6

Title: [Ruby] class diamond ($specific_collectors = ['TCPCollector', 'NetworkCollector'],){ - Pastebin.com (at pastebin.com)

antares_: running logstash-1.1.13-flatjar.jar

sagarv, looking now