#logstash

Those machines will actually be onsite at a facitliy

LS doesn't store data. it only forwards it.

Ahh, k

That was my bad ten

then

I thougth it did

It's ES that stores it?

Post filter?

yes, ES is one posisble output from LS.

k

Then nevermind

lol

Maybe I'll go with your method

haha

Just lsf->LS->ES on cloud

Well that makes it way easier than. :)

if your typeB serves wrote to your cloud ES, it sounds like you'd be in good shape.

yes, exactly.

Yep, didn't realize that.

The filters just need to live on the typeB machines, correct?

if you only have one LS layer (typeB), then that's where the filters would live.

that would be the one advantage of running an LS in the cloud, would be to have your rules centralized there, but...

Okay, nah, that's fine as they may need to be different per machine

Awesome, thanks for the help rastro, I think that is enough to get me going where I need

but with LS on typeB, you can have different rules per tenant, and only update a tenant's config when needed.

Yeah, I think I will go that route

good luck!

I guess I do have a follow up question

I've looked a little bit at datadog

so soon? :)

I'm guessing that's basically similiar to running ES in the cloud here

man, it seems like every regex / filter I add in my filters drops my processing rate by 3-4 KiB/s

What would be the advantage to me say going with a service like that? Just nothing for me to maintain?

xamox: i have too many hosts to think about datadog's pricing.

Okay, that's what I thought. I maybe in a similiar situation as there are a decent number of the type A machines feeding into type B

dyer: make sure your regexps are efficient and that you're using conditionals to make sure stuff is only running when needed.

Alright, cool, well thanks for all the help rastro

xamox: i have 500 machines in my POC, so...

yeah, good luck!

is this an efficient regex? ( if [host] =~ /127\.0\.0\.1/ { mutate { remove_field => "host" } } )

dyer: does host only contain the ip address, or more than that?

more

dyer: it's a simple literal match, which should be good.

dyer: do you have other fields on which this decision could be based?

that alone dropped my throughput 5k

rastro, okay, thanks for tossing out that number, not quite there yet but rolling a bunch of machines out and will probably getting not far from that within the next 6-12 months, which is why I'm looking at logstash as currently we aren't doing any log forwarding and need it to scale up

dyer: do you check the ip in [host] more than that once? it might make sense to grok the ip address out first.

_Bryan_ thanks, will try that out

Right now I am just starting to try to optimize my filters, so at this point I have just the following

very very basic

<http://tinyurl.com/q2jjmty>; (at gist.github.com)

For something like collectd, is it possible to tag metrics.. and have some be filtered one way, and others filtered another way?

dyer: assuming your real input isn't stdin, who is shipping the logs?

I am debugging crappy performance right now

so I am just cating to stdin

I have 2mil lines of production logs as my test case

my intent is to test filters one by one to find the shitty ones and fix them

dyer: but if you're going to be shipping from LSF, you can add a field *there* to identify haproxy/prism rather than scan [message].

dyer: do your logs have a standard prefix (like syslog's)?

no, unfortunately

dyer: so no initial date/time?

well we ship everything via syslog but the patterns are all over the place

but the transport from each DC is syslog to AWS, adn in AWS we process logs

the point i was trying to make is... if [message] has a common date/time (etc) string at the front, i would suggest grok{}ing that off first, and putting the rest back in [message].

then your regexps would run against a shorter string, which should make them more efficient.

Is there a way in logstash to buffer say 20 log lines, before I sent them off to output{} ?

j_t: some outputs have a buffer, like elasticsearch{}'s flush_size.

rastro: ok, yea I'm using the kafka output. I see it's an option now: https://github.com/joekiller/logstash-kafka

Title: joekiller/logstash-kafka · GitHub (at github.com)

rastro, so you would suggest I grok everythign first ?

dyer: the "better" the regexp and the shorter the field, the faster your =~ should go.

dyer: and the faster any subsequent grok{}s would go, too.

dyer: http://svops.com/blog/processing-common-event-i...

<http://tinyurl.com/nrw8oqa>; (at svops.com)

k, ty I will real it right now... one thing... I kinda inherited a lot of this so can you explain to me what these mean in a pattern

%{D}

and %{Q}

dyer: that should refer to a pattern named "D" or "Q". I don't think they're standard ones, so I'd look around for some custom patterns.

ah, I found them

## SHORTCUTS:

Q ("){1}

GD %{GREEDYDATA}

D %{DATA}

wow. your predecessor was lazy :)

Does anyone know if its possible to get Kibana to switch between dashboards on some interval? If you've ever used Ducksboard you'd know what i'm talking about

dyer: have you tuned LS? If not, you'll get a lot of return with that, too.

BlakeRG: haven't seen it. i had the browser do it for me.

rastro: how so/

bookmarklet?

BlakeRG: google found this one for chrome: http://www.crx4chrome.com/extensions/dlknooajie...

<http://tinyurl.com/nnhebjb>; (at www.crx4chrome.com)

BlakeRG: i forget what we used at the last company, but it was some extension like that.

nice rastro!

untergeek : thx for finding the relevent bug to the one I opened! I tried searching this morning but found nothing with my keywords.

<http://tinyurl.com/mbks7pj>; (at chrome.google.com)

rastro, I dont know... what knobs would you recommend us looking at ?

right now w/ my "production" filters I am getting 4KiB's :facepalm:

thats testing locally of course

https://gist.github.com/anonymous/003069a7c6673... .. is there a way I can cherry pick the keys I want from this json in logstash? Or can I drop certain keys?

probably worse in production using redis inputs and ES outputs

<http://tinyurl.com/p8rhtsg>; (at gist.github.com)

dyer: filter workers, output workers, etc. Some more info here: http://svops.com/blog/logstash-best-practices/

awesome stuff rastro thanks much

good luck, BlakeRG.

dyer: you can certainly remove fields. try mutate->remove_field.