#logstash

Title: Kibana3.0.1-ES1.1.1-issues - Pastebin.com (at pastebin.com)

config, error, and manual curl

in that pastebin

Oops! FacetPhaseExecutionException[Facet [0]: (value) field [shortterm] not found]

anyone see this before?

anyone have any insight / can point me at a good url for writting my own plugin (for output)

anyone have any insight - http://pastebin.com/p2kjvvjV getting the misconfigured proxy error

Title: Kibana3.0.1-ES1.1.1-issues - Pastebin.com (at pastebin.com)

modulusbass, use something like chrome where you can see network traffic..

(trying to find a guide)

any way to add host field or ip address to file input plugin for generic logs

modulusbass, basicly, use chrome, right click on page, Inspect Element -> Network Tab -> Refresh Page -- You'll see all the network requests and you can debug from there.

i'm just getting 0.0.0.0 I tried host even though it's not defined on the site as useable it didn't work

jason__, No, input plugins happen at the begining of the pipeline, so there is nothing to add (per se)

yeah i can see where my connection is failing :) (failed)

net::ERR_CONNECTION_TIMED_OUT

jusdt trying to figure out WHAT could be blocking it

ah ok

modulusbass, Using a proxy setting? and maybe the proxy can't access..

especially since i can call that url manually with curl

nope - no proxy, everything lives on the same box

do you get better performance by declaring multiple inputs/outputs in a single logstash conf, or running seperate logstash processes?

modulusbass, do you use a proxy to access the internet? If you do, it's possible the browser is trying to use that to connect to ES.

jason__: Does add_field do what you want?

patarr, it's threaded -- so basicly you're already running multiple processes.

(threaded where possible)

coolacid: so might as well just define them for one logstash process?

coolacid: hmmmm that just might be an issue.. checking

patarr, correct - I would suggest splitting out input -> filter -> output and putting in a message queue (like rabbitmq) so you cache events between steps..

does anyone know how to do collectd_type:memory to show up properly on the kibana histogram

coolacid: not sure what you mean

particularly the x axis in GB

coolacid: Thanks for the blog post about making your own MaxMind-compatible dat database

hi all, is there any way to have logstash ouput just the raw log line without JSON fields?

patarr, Logstash Collector -> (Output) RabbitMQ -> Logstash Filters -> RabbitMQ -> Logstash Indexer -> Elasticsearch

csd126, thanks!

waltertv: message_format => "%{message}"

not sure you mean something like add_field { host => "192.168.4.197" }

waltertv, which output... but you would want to use a codec => plain

thanks csd126

coolacid: wow really? Any reasons why?

csd126, that doesn't exist in some new versions and depending on the output.

Yeah, I should have asked what output he’s using

patarr, what happens if you need to change a filter config? You start loosing some events if they are syslog/udp/network

coolacid: i'm outputting to file

waltertv, then look at the codec setting

coolacid, i have the codec set to codec=> plain { charset => "UTF-8" }

waltertv, try "line"

waltertv, csd126 is correct also, looks like there is a message_format config on file as well - try his suggestion

jason__: Yes, isn't that what you want to do?

thanks, coolacid. appreciate it

CoolAcid - thanks - though not the EXACT problem you ponted me to the real issue - AWS security groups, I made the mistake/assumption that queries to ES went from the server running kibana NOT the client - once i added our office nat ip to the securioty group port 9200 all works

coolacid: why would I lose events?

yea, not working but I think it might be my logstash process

one sec

CoolAcid- so thank you very much for kicking me in the right direction

modulusbass, I'd suggest you look at https://github.com/coolacid/elasticsearch-proxy

Title: coolacid/elasticsearch-proxy · GitHub (at github.com)

It's rather your syntax.

Title: logstash - open source log management (at logstash.net)

modulusbass, although may not be the exact option, there are others listed there as well (or just a plain old proxy like https://github.com/elasticsearch/kibana/tree/ma... suggests)

<http://tinyurl.com/n9o98nx>; (at github.com)

illsci: I don't know what you mean. "1.4 tags are in ruby" ?

thanks coolacid - now that i have my POC set up - if anyone has any cool tips for collecting logs form all over AWS (VPC and non) im all ears :)

patarr, if you have Logstash Listening on a port, and you have to restart logstash what happens to the events while LS is restarting?

krm_: yeah, known bug, sorry abotu that. We're fixing it for the next release.

modulusbass, https://github.com/coolacid/GettingStartedWithELK

Title: coolacid/GettingStartedWithELK · GitHub (at github.com)

coolacid: they're sitting on the rabbit queue.

aha! yea my logstash process is acting weird on a new machine I just turned up

add_field => { host => "192.168.x.x" } worked

patarr, right (that's what I was suggesting.. use RabbitMQ inbetween processes)

coolacid: I have it like this. logstash sits on app machines, and just outputs without any processing to rabbitmq. Then the indexer machine grabs them off the queue using rabbitmq input, then filters with grok, then outputs to ES.

patarr, yep.. that would work fine.

jason__: Ah, even better! :)

=D

No statsd users here?

DanielHolth: what about it?

I can't get it to produce udp packets in logstash 1.4.0

DanielHolth: my setup works..

Care to paste?

hold on

whack: https://github.com/elasticsearch/logstash-forwa...

<http://tinyurl.com/l7a2b3c>; (at github.com)

I went to build the latest I could find today with an existing build job and just changed the branch and it failed and then I went to look at it... and its not go anymore?

DanielHolth: http://pastebin.com/Hvg69wzf

Title: statsd { type => "app" namespace => "s - Pastebin.com (at pastebin.com)

I was trying to match up version numbers before I figured out the right way to do that...

illsci: ignore that please

illsci: that was an accidental push to the wrong repo

I"ll delete it

there's no hidden or magic code anywhere. Master is the only thing that matters

I'll delete all the branches/tags

word :)

I was like.... why would he do that?