#logstash

Nevermind, start_position wasn't set to beginning.

Can a config dir be specified with logstash forwarder?

I'm using apache JSON logging... is there an easy way for me to then grab sub-strings out of request_uri and querystring? like grok on an event field?

`seen untergeek

semiosis: untergeek was last seen in #logstash 4 days, 7 hours, 14 minutes, and 59 seconds ago: <untergeek> if you have a value-list (cpu load 1 min, 5 min, 15 min) they will each be their own event

I LIVE!

hey, i have a question about high availability when using forwarder -> redis -> indexer -> ES. can anyone tell me, if the indexer tips over, is there any data retention or rescue for log events passing through the indexer at time of failure?

is there a need to setup statsd if logstash has a metrics filter now?

Does anyone know if I have only 1 and stop my single instance of elastic search and make a small config change and restart, does it reload all the data again hence duplicating?

phaze: it's less of a need now (imo), but eventually you'll want it

jerk: shouldn't - what sort of config change

jerk: also, if it's elasticsearch-specific and not really logstash-specific #elasticsearch might be a good place to check for additional insight.

just renamed my node_name

imperialwicket, good point. I'll check there.

I noticed there's a puppet-logstash, puppet-elasticsearch, but no puppet-kibana in the elasticsearch github

is that because kibana is that trivial to setup, or is it included in logstash somewhere

there's not really much to 'setup'. drop kibana in webserver folder and you're pretty much done

phaze: thanks

there are some minor changes you can make to the config file but i think i left it unchanged myself

roosri - there was a puppet-kibana for 2.0, i think it's still in electrical's github repo

i'd guess phaze has it right that either it's not happening or it's low priority since it's a pretty straightforward installation

tommy_o: not really out of the box. if you want to do that, you will need some sort of intermediary queue for events being "worked on", and then wipe them out of there when they're "done"

untergeek: idk about the gem path stuff. whack is the one to ask about that

yeah. He's off today on vacation

heh ok

it can wait until tomorrow

if it doesn't care about weird gem paths and the vendor path, then I'm cool with it

I kind of wished we'd done this in the first place

but, we didn't want duplicate gems and jars in our RPMs and such

so we designed it for distribution, not developers :(

i see

xBabyJesus: thanks for the reply. ill keep digging

trying to set up a very simple grok

here's my configuration

Title: input { file { path => "/var/log/syslog" type => "syslog" } } - Pastebin.com (at pastebin.com)

configtest is failing with the message "Error: Expected one of #, {, } at line 16, column 49 (byte 239) after output {\n elasticsearch {\n bind_host => \"elastic.mydomain.com\""

nvm

"," at the end of the line

I've been writing too much puppet

I'm not sure how to do this with the npm module: npm install brunch/brunch#6c8ed22

anyone every tried such a thing?

question: how is threading handled on file inputs?

I know I can increase filter workers

but is there a way to increase input workers

Lumberjack or NXlog for Windows hosts?

Whack: Do you have a minute?

Anyone around try passing a javaopt like ARGS="-Des.es.discovery.zen.ping.multicast.group='239.15.255.100' -Xmx$JAVAMEM -Xms$JAVAMEM -jar ${JARNAME} agent --config ${CONFIG_DIR} --log ${LOGFILE}"?