#logstash

i should make a diagram lol, this actually is extrememly difficult to describe

aspeer: if you're using LSF or LS as your shipper, you don't need redis. they will gracefully back off until LS is working again.

aspeer: i have ~500 machines running LSF.

aspeer: i do use redis for less-well-behaved data, like snmptraps and netflow, which don't back off...

We're at about 3200 that will be feeding into it with about 4.6TB

yeah, this is my POC :)

thats my problem, I've moved well pas the POC and my changes keep breaking crap lol

which its nice with multiple hosts it doesnt matter but this ssl thing is just stumping me

pun intended haha

does anyone know what happened to http://logstash.net/docs/1.1.9/inputs/amqp#sett... ?

<http://tinyurl.com/bj9uf7g>; (at logstash.net)

eh, lunchtime I'll wander back later with a diagram

aspeer: why do you need multiple certs?

I'm woring with 1.5.0b1, and the entire system seems to deadlock from time to time. No logged errors... just... <frozen>

working that is

any ideas or known problems?

I'm running tcp inputs, and statsd and redis outputs... nothing more.

dpp: high CPU while frozen? what's the max-jvm-size?

I'm not getting out of heap errors.. lemme check CPU. I didn't get alarms, set at 80%.

-Xmx2048m -Xss2048k

yeah, no crazy cpu spikes when they deadlocked

crazy high now that I'm processing the logs they died on

I have 3 servers in a loadbalanced setup, so turned on -v on one of them

you tried running vmstat?

maybe cpu wait or something

ummm.. unlikely in this setup, but I have the data.

:\ an LS profiler would be sweet

could show which filters are causing highest resource usage / processing time

yes!

or just export it to statsd/graphite!

chandlermelton: that's coming, not sure when though

ya thats one of those pie in the sky kinda things

the deadlock doesn't correspond to cpu/load. no cpu wait stats at all. no physical memory exhaustion.

do most of your logs go through the same filters? maybe one of them isnt playing nice in 1.5 and breaks when it gets that 1st log that uses it

dpp: how are you observing the freeze?

I send statsd messages as an output, and those counters stop growing. The process doesn't exit...

trying to confirm that we're not running out of sockets.

my current TCP connections count flatlines around then, but I have had it deadlock without that being the case as well.

dpp: so nothing is being output, that can mean many things

it oculd mean another output is stuck

sure. the other output is redis. I'm not specifying workers, so I assume just one thread.

it could mean nothing is coming into an input

"nothing being output" requires all outputs not be stuck and events actually coming into those outputs (From filters and inputs)

I know connections are being offered. What other visibility is available?

my one other clue is that three servers all choke at the same time.

have you tried the --debug switch?

I have not!

run LS from cli (not service) and add your conf w/ -f and add --debug

see where it stops

how much info does -v add to things?

idk

will run with debug next time it exits.

Does anyone have a good doc on how to setup a better map in Kibana?

I would like to show all the client ips in the better map.

jsandoval: well, you need to convert the ip-addr into geoip.coordinates

hrmm

I see, I thought kibanana did that for me.

guess not

Any good docs out there?

no, but logstash happily will :)

??

I am running ELK

whack, what insight do you have for figuring out where the messages stop?

jsandoval: within filter {} of the logstash config: http://pastie.org/9987917 you likely need to adapt the fieldname 'src_ip' and the path to the geoip-db

Title: #9987917 - Pastie (at pastie.org)

I'm using rabbitmq plugin, and I'm hoping there is a way to get access to the headers. does anyone have any idea?

dpp: well, I don't know much about your config or what the expected bheavior is, so it's hard to offer much specific advice

dpp: if you don't think any output is stuck, then it's probably your inputs not reading anything

what inputS? I don't know, what's your config? How freqwuently are events supposed to be coming?

so I'm trying to figure out grok stuff

we've got this silly log format that tries to be colorful

so, for example, when trying to say "meta" in yellow (i think..), it logs "\u001b[32mmeta\u001b[39m"

I'm trying to write a grok filter for that

and this isn't working: \u001b\[32m%{WORD:type}\\u001b\[39m

thoughts?

whack: I meant within the inputs themselves.

clearly you don't know what things I've done in my configuration.

but if I'm correct that something should be flowing in, I don't know what to poke at in the system next.

thefinn93: no double escape on the first \u? or just c+p miss?

kireevco: My 10-minute effort: http://svops.com/blog/better-than-newrelic/

Hello all !

double-p_: yeah, musta mis-copied

rastro: ha, cool!!!

rastro: thanks:)

kireevco: let me know if it needs more.