#logstash

i'm observing really strange behavior when adding more than 1 output to my logstash config

creating an example now

here we go: https://gist.github.com/drywheat/7b206a28cffdd0...

<http://tinyurl.com/nnjv2t2>; (at gist.github.com)

i have 2 outputs: kafka & riemann

everytime i run both together (even with guard conditionals) the throughput of my LS process drop significantly

yet, if i run 2 separate instances with the 2 respective outputs then it's perfectly fine

has anyone see this before?

my mapping template in my elasticsearch output doesnt seem to be doing anything

drywheat: if one output is congested, it will slow them both down. They're taking turns being congested, maybe?

drywheat: you are seeing that using two kafka outputs is better than 1?

mut3: it will only affect new indices.

i know

i actually derped, didnt set manage template to false

drywheat: have you looked at setting the number of worker threads to be greater?

thats probably it haha

the kafka output will have really low output in comparison to the riemann one

here's what's really odd, if i remove any outputs feeding the kafka ouput but still have it present then i still see blocking

let me show you

check that gist again for logstash-only-riemann-data.conf

so, nothing is feeding data to that kafka output but it still causes a blockage somehow

it just doesn't make any sense that an extra output would block if it isn't receiving any data

ya know?

should I see one directory for each index in the output from "curl -XGET http://localhost:9200/_aliases?pretty=true | grep logstash" in /var/lib/logstash/data/elasticsearch/nodes/0/indices/ ?

/var/lib/logstash/data/elasticsearch/nodes/0/indices/ shows 10 index directories and curl output shows some that aren't in that dir.

should they not be a 1-for-1 match?

does anybody know what is the oldlogstashjson codec?

there is no docs whatsoever

RobertDupont: it is still available as a plugin. it just doesn't have any configuration options

RobertDupont: looks like they changed schemas: https://github.com/elastic/logstash/blob/v1.4.2...

<http://tinyurl.com/oqzyams>; (at github.com)