#logstash

anyone available to help me troubleshoot elasticsearch output?

quarky: sup?

i think it might be my elastic search version, we're running logstash 1.1.10

and i'm not showing any data

i setup kibana and it too shows nothing being input into elasticsearch

however we just jumped up to 0.90.x on ES and I just read the docs saying it has to be 20.5

for 0.90 use elasticsearch_http

in the logstash conf?

yes

use that output

k let me try that

instead of elasticsearch it's elasticsearch_http and then all my existing elasticsearch options should take right?

umm, not sure, 1 sec

well, it depends on your options you have set, tbh

i cut it down to just the host and port atm

like, elasticsearch_http has no reason to care about the cluster name

it just needs 9200 for the port

k

just to verify

output {

elasticsearch_http {

host => "127.0.0.1"

port => "9200"

}

}

sorry to spam

also does ES log everything to STDOUT?

no

use gist/pastebin/whateverelse

k

okay so i enabled it, there was a warning about a temp dir

but kibana is still showing zero entries

do you have the elasticsearch-head plugin installed? install that if you don't have it.

it's very useful

no I don't let me take a look

does logstash log errors only?

quarky: by default yes

add --verbose and --debug flags if you want that kind of logging

k

can you verify this is correct for me?

Title: loggerconfig - Pastebin.com (at pastebin.com)

hey, whack

long time no irc

question:

if I'm trying to do metrics and a line has a grok field defined like this: %{WORD:method}

and a later field like: The method took %{INT:duration:int} ms.

is there a way I could rename duration to %{method}.duration ?

or would I need to do some fancy mutate work?

untergeek: good question!

it's for a statsd feed

i figued.

I want to track duration per method

sure

quarky: when you open up the elasticsearch-head plugin in your browser do you see indexes being created?

untergeek: i totally get why you're doing it

so, on this appliacne, they limit the core dump size to something small, so it creates a bunch of core dumps. but the scp doesn't allow globs.

lolwut?

er, bunch of core dump files

so the head plugin is installed

but not able to get to it due to some environment issues I think

sec

don't forge tthe trailing slash

http://localhost:9200/_plugin/head/ <- need this

http://localhost:9200/_plugin/head <- no work

is weird

lol

well we'are also not in a vpc

so i'm having to port forward to test

vpc?

oh, virtual

pc

a vm.

me either.

vpc = virtual portchannel

no, AWS

i have to bounce through another box

ah

just socks5 to the first box, then ssh to the second using that socks

(assumping putty anyways)

lol well it's not working out

I need some help converting from an 8601 date to unix epoch so I can reliable search log entries.

Any ideas?

so what would cause logstash not to output to elasticsearch

if i'm using http

have you tried a stdout output just to make sure it's getting that far?

yeah let me try it again

i'm running this through a package so i don't have console output

Say i'm forwarding logs to my logstash server from a windows host, via Snare, Snare forwards them in syslog format. So to use the Eventlog plugin, do i just edit the logstash.conf on my logstash server, and add another input? or do i need to do something different that Snare on the windows host?

personally, i'd send them to a syslog server, write them to file, then use logstash to pick them up

quarky: not sure what you mean, entirely, i guess. might try just running it using java -jar ... to work thorough your issues

so send them to a different server? not sure what you mean by use logstash to pick them up, I thought they all had to end up on the one server?

well, it doesn't *have* to be logstash

so, you can do windows -> syslog -> disk -> shipper -> redis -> logstash -> es

wherein a shipper is logstash/beaver

umm. there's a few others

i see

okay so STDOUT shows the data coming from one of the clients

it's just piping that output into the logstash log

so either logstash isn't writing to elasticsearch, or kibana is jacked up

okay

hello logstashers, I have questions :)

I want to perform a mutate

to remove a field

so, did you get into the elasticsearch-head plugin to see if the index is being created?

if it has a specific value

match => [ "to-host-dns", "127\.0\.0\.1" ]

remove => [ "to-host-dns" ]

(you could do it from cli too, but i lack desire to look it up atm)

head is not playing well with the forwarding atm

but, unfortunately, mutate does not support match :(