#logstash

hi

any experience on setting ELK for a rails app?

Whats the easiest way to apply a grok pattern on a field and create a new one from a capture group?

rails/postgres/redis

Hey, is anyone doing anything to monitor data loss within Logstash pipelines?

what do you mean by data loss?

as in if Logstash is struggling to write data to the output for a give reason, I was under the impression Logstash will drop the data

i think if the output can't write it will log an error. I only use stdout/file/elasticsearch and if ES can't be written to for example you should receive an error log

so as for monitoring this, in the past i have been using a combination of tailing the log file and scanning for errors that i then e.g. send as emails and/or use a metrics soltion (e.g. graphite) to report

so i reckon it would be less data-loss monitoring but rather general error monitoring, where in turn you could differentiate between the meaning of those

for example my local logstash does this when my elasticsearch is down: http://pastebin.com/5RSPzCKc

Title: Attempted to send a bulk request to Elasticsearch configured at '["http://127.0. - Pastebin.com (at pastebin.com)