#logstash

Ga This stupid upgrade gone wrong.

Any ideas for updating kibana? I have seen a bunch of fail.

Finally reverted to something that works.

Also upgrade of ES broke indexes/field references. Any pointers on that would be appreciated.

has anyone here done any pruning (like with curator.py) with different pruning schedules based on different indexes?

like keep stuff stored in index type a for 30 days but this really chatty stuff that is index b only keep for like 5 days

cully: I use https://github.com/imperialwicket/elasticsearch...

<http://tinyurl.com/qh35m9m>; (at github.com)

cully: You may also want to look at http://logstash.net/docs/1.3.3/outputs/elastics... I think you can setup a TTL in the template for documents. But that may be a bit harder to figure out from my experience.

<http://tinyurl.com/l97nz5p>; (at logstash.net)

for the volume that I'm looking at TTL is too resource intensive; but I'm definitely looking at that index-mgmt stuff - thanks!

untergeek: so, did some playing around w/ using ES as a KV store... 3 nodes (5 shards; 1 replica) using mget, I can lookup 1000 unique keys in < 200ms.

cool

so now they want to replace DynamoDB w/ ES :)

woo!

well done

are the ES nodes local or in AWS?

AWS

I allocated 8g heap on an m1.xlarge

and it's cheaper and more performant the way you've described?

than DynamoDB?

hello people

certainly more performant, and we absolutely believe it will be cheaper

atgiant: we don't have people here... just bots

hello bots :)

resistance is futile

untergeek: but the biggest win is that it will be more versitile; We can use the same infrastructure to replace both DynamoDB *and* CloudSearch (further increasing the cost savings)

hey folks, quick question I'm hoping. Can I configure one lumberjack instance to send logs to different networks from one config file or can it run multiple config file? Or would I need to run additional instances for each config?

as an example, I'd like to run both of these configs from one instance: {

"network":

{

"servers": [ "logserver:5545" ],

"ssl ca" : "/opt/logstash-forwarder/logstash.pub",

"timeout": 15

},

"files":

[

{ "paths": [ "/var/log/apache2/apache*.log"], "fields": { "type":"apache", "environment":"staging" } },

{ "paths": [ "/home/sites/api/logs/global.log"], "fields": { "type":"app", "environment":"staging" } }

]

}

{

I'm trying to do this for two reasons. 1) I may want to send different kinds of logs to differnt logstash servers and 2) when sending to one logstash server it seems far easier to configure the logstash side of things if it gets different kinds of logs to different ports.

van_jwc: pastebin!

pastebin?

Title: Pastebin.com - #1 paste tool since 2002! (at pastebin.com)

don't paste large amounts of text to the channel, use a paste bin of sorts

ah ok. sorry. I havent used IRC for something like 10 years before today :P

if the channel were busier it would make a lot more intuitive sense :)

that being said, I don't know anything about lumberjack, so I can't help you :(

van_jwc: Then you'd be expecting to get DoS'd off the channel for pasting that much text :D

ok, sample config now at: http://pastebin.com/GY7SxjQm

Title: sample config - Pastebin.com (at pastebin.com)

I haven't used IRC in like 15. Ever since my emulator coding days :)

thanks for the pastebin tip!

gah, just hit a bug in the 1.4.0 beta 1. when's beta 2 out ? ;-)

github made it look like 'soon'