#logstash

i can tail that file and see the logs being appended to it

jamesturnbull: eh, not happy with this implementation, patch shortly.

jamesturnbull: https://gist.github.com/jordansissel/2d9551befd...

<http://tinyurl.com/l9skdhw>; (at gist.github.com)

logstashbot: ping

pong

whack: any bot gripes/requests?

semiosis: none at the moment! I think it does just fine :)

jamesturnbull: same q?

whack: <3 glad to hear

whack: i'll be in bay area mid april for red hat summit, we should meet up!

semiosis: woo! Yes.

I was going to submit a talk to devnation but I ran out of time

(I think those coincide?)

whack: it's still open, rolling cfp til feb

i just sent mine in yesterday

oh really?

well shit then

I had no idea!

will submit!

hope to see you there

evening folks!

avleen: avleen avleen avleen av

whack does this look right? http://sebsauvage.net/paste/?15fa1612422eeffd#D...

<http://tinyurl.com/pst3t9h>; (at sebsauvage.net)

whack whack whack whack whwaaaaaaaaaaccccccccckkkkkkkkk! :)

semiosis: I am fine

LuxuryMode: yeah seems OK, is this file an existing log you want to import?

not sure what you're asking, sorry

LuxuryMode: no worries, I'll rephrase

LuxuryMode: how big is this file?

i just want to feed logstash with the logs that are going in there

small file

do you want the whole file, as is, imported?

or just new events?

whack: i haven't had a chance to test the grok patch yet. hopefully tomorrow now :)

jamesturnbull: keep up the good work!

avleen: I haven't done much on it yet, still need to make sure the tests pass

whack 82k currently

avleen: but I think I can get grok to be a bit faster

LuxuryMode: is this on windows?

hellz no

path => '\tmp\application.log'

lol :D

make that path => "/tmp/application.log"

I think the backslash \ is tricking it

oh shit good point

wth did i do

jamesturnbull: thoughts on the output? https://gist.github.com/jordansissel/2d9551befd...

<http://tinyurl.com/l9skdhw>; (at gist.github.com)

LuxuryMode: hah

jamesturnbull: have you used statsd count before? would it be as basic as this, if all i want to do is count the number of hashes that are the same?

LuxuryMode: sometimes the brain makes you do things without you knowing, then you spend an hour trying to figure out what the hell is going on!

count => [ "%{hash}"]

whack: i did some tidying up of my config today, removed some unnecessary mutates and prunes. My apache access log was only ~1hour behind at peak time. that's a marked improvement over before. still not certain where the new bottleneck is, but I'll find it ;-)

avleen: do let me know if you find moar bottlenecks

whack works now, thanks a lot

avleen: you *should* be able to test throughput with the generator input, if that helps

LuxuryMode: high five!

now to figure out a sane logging format

LuxuryMode: glad it was something simple to fix ;)

yeah me too

logstash is very friendly

so easy to set up

now to figure out what kind of EC2 instance i really need for this. and what kind of volume should be attached

whack: u experrt! s/\\/\//

so for now i just took an existing AMI and launched that

whack: generator goes damn fast, as it should :D my hunch right now is that it's a lot of work to just parse my logs. each apache log line is ~3k, and has 33 fields, so you can imagine the regex for grok is pretty big. that's why i'm also really keen to try this patch

anyone know how to change the username and password that kibana uses?

rystic: try the metrics plugin

whack: oh, a question actually. if i drop a plugin in my plugins dir, which has the same name as a built in plugin.. could i expect mine to take precidence over the built in one?

avleen: I mean generator -> your filters

avleen: your plugin should override an existing one

becuase --pluginpath prepends to the RUBYLIB directory list

avleen: would you be open to trying json logs in apache?

avleen: would remove any need to use grok for this specific case

save yourself tons of CPU

,,(json apache)

semiosis: Error: "json" is not a valid command.

fail!

,,(apache json)

semiosis: http://untergeek.com/2013/09/11/getting-apache-...

100% yes. Once these apache logs are migrated over fully to logstash, that's my intention :)

probably also fail

avleen: you can do it simultaneously, btw

since apache can output logs to multiple places

well, i suppose i could write the file out twice

yeah

:D JINX!

I mean, whatever works ;)

just saying you don't have to do a cold cutover

I'll make a ticket formyself to try that next monday or wednesday.

i know my logstash boxes aren't anywhere neat 100% CPU

it's 20%-30% CPU at most

avleen: make it monday!

avleen: interesting

but.. that doesn't mean they aren't spinning and context switching a lot, etc

avleen: can you take a sample log using the generator input, through your filters, and use stdout output? try to measure peak throughput?

semiosis: dude, if i could, i would! I might be able to. it's possible.

whack: I'll do it in about 5 mins, let's see what happens.

jamesturnbull: will metrics work for my use case? i want to find unique hashes over time. e.g. in the last 7 days, i want to find hashes that have the lowest counts

avleen: woot

avleen: I measure throughput (poorly) by doing: generator -> filters -> output { stdout { codec => dots } }

avleen: piped to: | pv -War > /dev/null

dude i love that

one dot == one event, pv outputs bytes/sec, ergo, events per second.

hacky, but it's a beautiful thing.

avleen: in your case, use same filterworker count as you do today

avleen: you'll probabaly want 2-4 generator threads: input { generator { threads => 4 message => "your sample apache log" } }

and *maybe* more than 1 stdout worker, but I don't know

whack: not expecting anything, but just on the off-chance, know if anyone uses varnishlog input or cloudwatch output? :)

I've seen folks file bugs or ask questions about varnishlog input

cloudwatch I don't know

bugs against varnishlog?!

but it's perfect!!!

lol j/k

haha

hmm, can't find the bug now

maybe it wasn't a jira

whack: gh issues disabled

@jira

`jira

semiosis: https://logstash.jira.com/secure/Dashboard.jspa

[logstash-users] Logstash throughput issue when shipping Varnish logs

2 months ago

seems like the logstash debian package is 75MB, the one from here: http://www.elasticsearch.org/blog/apt-and-yum-r...

<http://tinyurl.com/nyg99vr>; (at www.elasticsearch.org)

avleen: kv not good enough for auditd logs?

barthoda: that is correct.

barthoda: logstash releases average 60-80mb