#logstash

I use lumberjack which handles the same functionality that logstash uses redis for

whack: so my setup looks like this: 30 prod servers -> rabbitMQ -> logstash filtering server -> ES clusters for certain logs types -> back to logstash server hosting Kibanas

aaand HR just told me they scheduled a meeting for now 30 minutes ago.

i was thinkig to put redis on all out web servers....and ship logs to local, and tell elastic to get there for input

bbl.

whack: do your servers send logs to a particular hostname, or some load balancer?

imperialwicket: lumberjack handles that; I give a list of servers and it figures it out

that works fine but juniper kit tends to not run lumberjack to well

well certainly lumberjack isn't a solution for all situations ;)

thought there was a magic trick I was missing - damn !

most juniper stuff is freebsd-based, isn't it?

yeah

i can get to shell but the memory is not massive for java stuff to run

lumberjack should run on it then, though I haven't tested.

lumberjack is written in C, not java.

I think I found my strange message problem… a couple of extra escapes in something that's getting single quoted rather than double quoted.

also i think it might violate our support contracts :)

worth a go though

eper: indeed, which means you should get logs from whatever supported interface there is.

MixMuffins: I haven't read everything, but have we addressed your grok filters?

whack: but it's still the case that if you want to deploy an 8th LS/ES server, you need to push that updated server list to all the shipping servers before 8 gets used, right?

imperialwicket: well, yes, but at some point to add another server to any system you have to have a way of advertising that additional server's presence.

MixMuffins: oh, you're gone. Well, if we haven't already addressed, use grok less. Grok is great, but if you can avoid it, you're going to get much better performance

imperialwicket: and the config file is automatically generated for lumberjack, so there's nothing to do by hand for me

(in my case, anyway)

MixMuffins: as an example, I went from shipping logs as strings, through grok, to shipping as json objects with minimal groking, and went from 350e/s per LS node, to 1800+

whack: sounds right, just checking. thanks for details, much appreciated

can someone tell me what is the best INPUT for redis, Thread and batch_count

there's no best

JohnS_: experimentation is the best recommendation

rashidkpc: yeah, actually. The majority of my logs aren't using grok filters; they're actually being input directly thanks to changes using a logstash json appender that formats the logs into an easy-to-parse format for logstash

So, short meeting, but I need to head home for a bit. One of my coworkers passed away this morning, which is a bit of a shakeup.

whatc: yes but there should be some guidance

MixMuffins: my condolences

as per cpu or somthing else

JohnS_: I don't really have any guidance. If I did, I'd set that guidance to be the default values.

so my guidance is generally "use the defaults"

whack rashidkpc: Thank you for help though. :) I'll have to talk with you guys on a different date. I seem to have the worst luck so far as coming in here for questions; gotta love unexpected events. In any case, have a wonderful evening.

there should be something about on how big rate can logstash read from redis

JohnS_: taht depends on your configuration (software and hardware)

I've done 25000 events/sec with logstash 1.1.13 on my workstation

whack: yes but with 4cpu machinge what threads and batch to set?

If you care, here's the results of my last test set

Title: semicomplete (at docs.google.com)

wow

you're most interested in rows 10-13

i assume thats rabbitmq with the default settings?

rashidkpc: the plugins with default settings? yes

I've had no problem consuming 6500+ e/s out of rabbit with ack disable

disabled

nod

and thats annecdotal

I was testing performance scenarios where acknowledgemnets were required

lah

ah

(lumberjack and redis can do them, rabbitmq can but has a massive performance hit)

you can do acks + prefetch = 50 and get OK perf I think

but it's still lots of acks

lumberjack ammortizes ack costs by batching it like tcp does

whach: one last Q

how do you do the rate tests?

hackishly

lol

generator input (w/ count set if I want a finite test)

or generator input + metrics filter + stdout output

or generator input + stdout output w/ debug_format => dots, piped to 'pv' to gauge event rate

clever

or just stout with time and wc-l

whack: aside doing a load(__FILE__) inside my plugin, is there anything else I might need to do?

because I can see my code calling load, but new method definitions don't take effect

:(

I've never bothered with code reloading in ruby, so I don't know if it's more complicated than that.

well, seemingly it is

unless jruby does something different to ruby

well jruby does all sorts of caching optimizations, I don't know if the load() will invalidate the cache or whatnot

are you loading code from the jar? or local files?

maybe ask in #jruby

afk for a bit

plugin files come from the filesystem (local files), the rest of logstash comes from the jar

aw yeah, did someone say editable filters?

<http://goo.gl/sG8ooh>; (at dl.dropboxusercontent.com)

Whoop: ^

pretty

Sweeet

rashidkpc: is that in head?

just about to merge it

awesome :)

will upgrade tomorrow, thanks!

merged

Whoop: note that it applies to all filter type except time. Time is sort of special.

types

time is fine :)

thanks!

events are taking 3+ minutes to appear in elasticsearch, how can i determine if logstash or elasticsearch is slow?

a13x2121: are you using some sort of queuing mechanism?

not currently, client syslog ->syslog (logstash) to elasticsearch

are ES and LS on the same box? Using embedded ES or standalone?