#logstash

warkolm would it make sense to have an architecture similar to this setup, but with a file output for the shipper instead of redis? (https://www.korekontrol.eu/blog/tips-for-centra...)

<http://tinyurl.com/lwwjcdl>; (at www.korekontrol.eu)

it seems like extra work to me

why ship from a file to LSF to LS to a file then to LS then to ES?

why not use rsyslog to send to a central instance, then let LS process that

but ultimately it's up to you

because I want to run filters > ES, but no filters > archive.

then use the central rsyslog as your archive

then you don't even have to worry about pushing the data through LS

through LS or through LSF? I'd like to use LS's filtering language to get the data into elasticsearch. Some of our syslog data is awful, and needs work before it can be useful.

Mixologic: I think he's saying use both. rsyslog for your originals and logstash for the features you want.

jabroney1: yeah, thats what I was confirming rsyslog to do the transport and storage on the loghost (replacing our current syslog-ng), then logstash read from files, as it does now.

lol

then yeah, we are talking about the same thing

cool. I wasnt sure if LSF would work better than rsyslog and RELP, or be a replacement for it.

it's an alternative

Mixologic: +1 what warkolm is saying

and sounds like that rsyslog would be the easier alternative.

has anyone got email with with logstash

im trying to catch certain alerts and send an email out

?

you can use rsyslog instead of logstash-forwarder if you: 1) already know rsyslog, 2) don't need security, or 3) already have rsyslog doing something else even if you don't know it but dont' want to install anything else

whack: we do need security (some aws instances sending back data to our infra) but figured this would do the trick: http://www.rsyslog.com/doc/rsyslog_tls.html

whack but we really dont know rsyslog any more than we know logstash-forwarder.

I don't remember how rsyslog behaves under network problems with the relp protocol, but it's something you'll want to research

lsf focuses on secure, reliable, bandwidth-friendly transport of data

whack: my concern is that lsf would be reformatting the data to put it into logstash - ie. making it json before we wanted it to.

we've got stupid things in our syslogs, like carriage returns in the middle of fields, so ideally it would just do straight transport if we wanted.

Mixologic: lsf doesn't reformat anything

but it is line oriented (as is rsyslog)

lsf also only delimites messages on line feeds, not carriage returns.

which is perhaps not the best behavior given osx (\r), windows (\r\n), and unix (\n) systems all use different line terminators

thought osx switched to \n?

jkitchen: it depends on the program :(

some programs use \n, some use \r still

FileMaker Pro, for example, still uses \r

weak

hey guys hopefully a quick q. trying to take log records from elasticsearch through logstash to perform geoip. I need it to replace documents. Whats the ES output's 'document_id' field?

something like document_id => "%{_id}" ?

that is, in the logstash config

maxim111: I don't know if that's possible in logstash 1.4.2

but we added support for it for 1.5.0 beta1 I think

ok thats too bad. so basically the elasticsearch input in logstash doesn't expose the document _id to be used as a variable?

maxim111: in 1.4.2 I don't think so, as said, in 1.5 I'm pretty sure we fixed that

ok cool. i might try the beta then. do you know how to refer to that variable? "%{_id}" like i wrote above?

<http://tinyurl.com/outaokd>; (at github.com)

maxim111: in 1.5.x it's probably [@metadata][_id]

so %{[@metadata][_id]} in a formatted string

I *think*

this is recited from memory and glancing at the code

looks about right per the code. thanks a lot

is there a logstash beta available for download that includes all the various addons that the stable version has?

there is a 1.5 beta with all the standard plugins

mind linking me? i'm having a hard time finding it

<http://tinyurl.com/pdltdyg>; (at www.elasticsearch.org)

wonderful thx

doesn't seem like that beta build has all the plugins as it fails with my config file. doesn't look like it's built the same way as the stable release. i'll just wait until it's released i guess

what plugins

mind you with 1.5 there is a change to the way plugins work. you can install things individually now rather than all or nothing

i blew up my logstash server accidentally a while ago due to lack of proper documentation :(

thought i was blowing up another server, but nope

going to reinstall it when 1.5 comes out :D