#logstash

Hi. What's the recommended way of sending auditd logs to logstash given that they are binary?

Is there a way to take an elasticsearch index and enrich it's documents with data from a second index and output it to a third index? I was assuming the elasticsearch logstash filter plugin would do that but it seems to query against previous events.

Ahuge: logstash has an elasticsearch{} input that will read existing documents. Depending on what you're doing, the translate{} filter might also be of interest.

what mechanism does logstash use for it's own logging? does it use log4j?

no

it uses a rubygem whack wrote called "cabin"

rasto I'm using that to populate my event with the original document but now I need to query data from a seperate index based on a field in my original document. In this case it is a custom id.

Ahuge: if you're in the middle of processing a document, then you can use the elasticsearch filter to add data it to from a previously-indexed document.

rastro: Wait so is the elasticsearch filter pulling from previously queried documents in memory? Or is it doing an ES query like the input plugin is?

Ahuge: both query from ES.

Ahuge: LS is not caching your documents in RAM.

rastro: k good haha

certainly not in any format that would support ES queries!

torrancew: thanks, and where can I find it's own logs?I mean where it write it's own logs and where I should set the path that I wanted it to write it's own logs?

rastro: Does this conf look like it should be correct? http://pastebin.com/pMFUG9ip

Title: input { elasticsearch { hosts => "es_server" index => "raw_index" - Pastebin.com (at pastebin.com)

Ahuge: i haven't done that type of stuff, but the concept looks ok.

ladan91: by default, stdout, but the init script (and its associated config files) typically override that

rastro: hmmm alright thanks for your help. Ill keep tooling around and see if I can figure it out

torrancew: thanks a lot

Ahuge: is some part of it not working? that's not what you asked...

rastro: oh sorry my mistake, yeah I am getting "Failed to query elasticsearch for previous event" with the error :error=>#<NoMethodError: undefined method `[]' for nil:NilClass>, :level=>:warn}

can someone point me to the right direction? im testing out collecd plugin - https://www.elastic.co/guide/en/logstash/curren... - i can see that data is coming into logstash, but its not output into elasticsearch

<http://tinyurl.com/ot9xuwl>; (at www.elastic.co)

Ahuge: i would see if the input{} is working as you expect. Try removing the filter{} section and the elasticsearhc{} output and see if you good stuff on stdout.

rastro: alright will do thanks

is there any way that I can send logstash logs itself to logstash?or send them to rsyslog?

ladan91: i run a shipper on my ELK machines to send their logs to another server that runs a small but complete stack.

sending to its own stack is almost /definitely/ a bad idea

s/almost//

:) thank you

lol

it's a really fantastic way to piss yourself off, though

I say, having done it once before, but with LSF, not LS

"hey, i'll just check the, um... crap!"

yup

Also, LSF starts logging failures like crazy

which, if your problem was capacity, just makes it all worse

:)

hmm.

rastro: are you alive? :)

trying to move my elasticsearch off my logstash box, but logstash instance has deided it doesn't want to talk to elasticsearch, specificying a host with the ip in host => ["x.x.x.x] formatany ideas?

andrew[andrboot]: rastro could be a computer, in which case the answer was /never/ "yes" :)

heh.

my friend eliza has the afternoon off.

andrew[andrboot]: elasticsearch bound to * or ::?

haha