#logstash

funix: you want convert => { "[@metadata][rate]" => "integer" }

whack: I have tried that to . it give me same err - Exception in filterworker {"exception"=>#<NoMethodError: undefined method `>' for nil:NilClass>,

whack: codec => line {format => "rate2: %{[@metadata][rate]}"} in output shows the rate as x.x and not as an integer.

whack: config file - http://pastebin.com/NdjjABy0

Title: logstash rate limiting - Pastebin.com (at pastebin.com)

you need to separate your mutates

the ordering is confusing, sorry for that

add_field is run _last_

so after your convert

separate your convert into a new mutate { } and it should work

good evening/morning

i need an english to english translation. some fuqnut named cornelius van lunt seems to imply there are two elastic search masters with different cluster names... but there is only one. what gives?

WARNING: [logstash-localhost-76881-13456] received cluster state from [[Cornelius van Lunt][ZC5vyE2LRAieGVaAofOjwQ][localhost][inet[/192.168.134.17:9300]]] which is also master but with a different cluster name [Cluster [1ba07c0a-9b8d-6e50-ce7d-9cf8e6b4ebd4]]

whack: that worked like a charm. But now I think i cannot use it for rate limiting. Basically with my logstash config i compute drop % only when I have metrics filter applied which happens only once in x seconds, (flush_interval => 1). So I cannot apply drop {} to all the other events. So ideally I would have to store this drop % value in some global variable which only gets updated when I have metrics filter applied to find rate. Makes sense

?

So in short I would like to compute rate of events every x seconds, compute by rate limiting percentage and then apply it using drop {} to all the forthcoming events. After x seconds compute the rate and update my rate limiting percentage accordingly .

do any of you know what this means?

Jul 30, 2015 1:51:32 AM org.elasticsearch.discovery.zen.ZenDiscovery handleNewClusterStateFromMaster

WARNING: [logstash-localhost-77573-13456] received cluster state from [[192.168.134.17][AearZQ34T0GXl4meKCxs3A][localhost][inet[/192.168.134.17:9300]]] which is also master but with a different cluster name [Cluster [1ba07c0a-9b8d-6e50-ce7d-9cf8e6b4ebd4]]

it is a cluster of one... seems like it found two masters?

funix: yeah you can't easily do that; maybe a custom filter or ruby filter could help

whack: i was looking at ruby filter. Din't see good examples. Can you store a global variable using it which applies to all the events ?

If I use - filter {ruby {code => "$drop_per = 50"}} , how can i access this variable in other filters ? %{drop_per} does not work .

yeah, though it's tricky to do in a thread safe way, but you can use $foo for global variables

or @foo for instance variables (local to the ruby filter)

so you could do like

in the ruby filter, if the event is a metric, store the rate, if it's not a metric, choose to drop based on the rate

global var swould let you share data across multiple filters, but it introduces race conditions that make it not so safe

if I use $foo or @foo , how to access them in the drop filter ?

you can cancel them from your ruby filter

whack : easier way may be to do this externally fetching index statistics or something using a cron, compute rate limiting and set it to environment variable using export DROP_PERCENTAGE = <num> and then use "add_field_from_env" for every event in logstash to access this env variable and give it to drop {} ?