#logstash

dont think so, because loglines belonging together can potentially be days apart..

ah I see.

electrical: still having software raid performance issues? when I setup my 9TB software raid I added "dev.raid.speed_limit_max = 500000" to /etc/sysctrl.conf and also let it synch the drives with the partition unmounted. when it was mounted it was synincg at like, 30k/s, and unmounted it was syincing at 300MB/s

cd /Users/khanr ;

doh

:)

rashidkpc: in kibana 3, if the graph is stacked, could the tooltips such that they show the height for that color, not the total y value?

ralphm: -maybe-, I'm looking into it

:-)

rashidkpc: thanks for the awesome work

Any plans for scribe input ?

not that im aware off.

I like the fact that scribe stores data locally when it cant stream..

does anyone still use scribe?

Any other system capable of same?

I believe there is a thing for having a scribe server also push to redis

ok..

Title: comfirm/scribe-redis · GitHub (at github.com)

yeah

3/last whack

mmm lunch

What is the thing with having redis in the middle btw?

Dont really see.. just the fast speed?

tziOm: lumberjack does the same

the problem with scribe is that (iirc) the only way to give it logs is to use thrift, so there's no avenue for existing software and systems where you have no control over the code.

whack, same as in scribe->redis<-logstash->elastic ?

tziOm: lumberjack -> logstash

lumberjack pauses if logstash fails

whack, ah.. I was thinking lumberjack was some guy..

haha

hah

whack should dress as a lumberjack at all meetups from now on

hah

so I'm guessing if I use the elasticsearch_http option I don't have to set up logstash as an ES node like you do with the standard es output plugin?

hmm.. looks ok this lumberjack

rayrod2030: There's no "setup" to do so I'm not sure what you mean.

What is the reasoning behind having redis as a "man in the middle" for messages?

is it used as a buffer?

tziOm: buffer/router

well running stand alone with a previous cluster I had some trouble establishing communication until I opened 9300 in both directions between logstash and ES.

allows you to restart readers (logstash) without having writers recognize that

it makes implementing some message flows a bit easier

rayrod2030: ahh, yes.

whack, some examples here would be useful..

rayrod2030: with elasticsearch_http you only need to open access outbound from logstash to inbound on elasticsearch port 9200

tziOm: not sure what you mean?

yeah, i'm trying to decide if i need redis right now. in my setup i write all my syslogs to disk, and i have logstash readig from disk - so the disk is the buffer. not seeing any performance advantage or reliability advantage for redis, in this configuration

werd. that's what I thought. great!

tziOm: logstsah treats redis like a queue

whack, "making some message flows abit easiert"

tziOm: if your redis inputs are overloaded on a resource, you can add more logstash agents reading from that redis input to achieve higher throughput

yeah..

no lumberjack with redis though right?

seems...

<http://goo.gl/Otq8k>; (at twitter.com)

snap

didn't know lumberjack could do that already :-)

not sure it's in main.

readme still says redis won't do encrypted or compressed from lumberjack

rayrod2030: not implemented yet

requires features in logstash that aren't available yet

New news from newjiraissues: Jordan Sissel created LOGSTASH-1114 - Implement new event schema <https://logstash.jira.com/browse/LOGSTASH-1114>;

well, that's always fun. rebooted the vm and the config file works now...

whack: sorry it was a dumb problem. thank you for sanity checking the config and commandline...

hah, no worries :)

easier to pair up on crazy problems ;)

stupid computers....

im gonna watch some tv and relax a bit. catch you all tomorrow.

well, it works - but only when stdout/debug output is selected