#logstash

is _grokparsefailure added when *any* grok fails to match, or only when they all fail to match?

avleen: when all, in theory, except in some cases where there's a known-ish bug

but the intent is when all of them fail

ok :) cool, that's what i was hoping.

oh that leads me to another question: if i specify a type on an input, and the incoming data from lumberjack has a different type specified... what happens?

because all inputs need types

you're using lumberjack input w/ format set?

avleen: it's a misdesign that inputs require 'type' to be set

i'm hpoing to

json all the logs, etc

:)

avleen: so what happens is that if you have format => json_event, the 'type' in your event itself will remain

eeeeexcellent

at that point the 'type' on the input setting becomes more of a default-if-not-present

ok :)

yeah my logs aren't JSON at all yet

json is stupid, unformatted plain text for all!

NATURAL LANGUAGE WILL SOLVE THIS

:-D

yaml.

yaml's starting to grow on me ever so slightly.

like a fungus.

it's definitely not for a streaming format though :)

whack: would you mind scrolling up a bit for my coordinate array question? :)

I can't get it to not be a string in ES

well, you're setting it as a string

but it probably wants "coordinates": "-122...,37..."

msgpack!

instead of an array with 1 string, have it a string

avleen: lol

add_field => [ "coordinates", "%{geoip.longitude},%{geoip.latitude}" ]

how do I specify that without the quotes?

without quotes?

not sure what you mean

you want to have 'coordinates' be an array of two numbers?

like "coordinates": [ "-122

erm

like "coordinates": [ "-122", "37" ]

?

whack: hey

antares_: howdy!

whack: I have a couple of questions about developing plugins for 1.1.x

(backporting the new rabbitmq plugin as we speak)

I'm getting :exception=>#<NameError: uninitialized constant LogStash::Pipeline when I try to reference LogStash::Pipeline::ShutdownSignal

blerg, lemme see what I did there

In master it's just LogStash::ShutdownSignal, I think

yeah it's just that

I'm trying to move exceptions to a single namespace

whack: is there a way to run irb with all logstash dependencies on $LOAD_PATH?

bin/logstash irb

or pry if you prefer pry

irb/pry launched with bin/logstash (or java -jar logstash.jar pry) doesn't have *everything* loaded, but you can require "logstash/whatever" and it'll work

basically just sets up the right gem/lib paths

whack: yup, cannot access LogStash::Pipeline there either

require "logstash/pipeline"

hm, if I explicitly require "logstash/pipeline", I can

I should update the pry/irb stuff to load common things

should I do that from the plugin?

that seems a little odd

what do you need from the pipeline class?

maybe I am confused

are you working on the 1.1.14 amqp/rabbitmq stuff?

Hi, guys, I am using the embedded elasticsearch output, how can I reset the elasticsearch index quickly?

whack: sorry, I'm back. Will this actually work? add_field => [ "coordinates", %{geoip.longitude},%{geoip.latitude}]

I thought the quotes were mandatory around parameters

whack: yup

whack: I need the ShutdownSignal exception

otherwise I cannot cancel a blocking consumer on HotBunnies

New news from newjiraissues: Christopher Carpenter created LOGSTASH-1228 - UndefinedConversionError with EventLog and GELF <https://logstash.jira.com/browse/LOGSTASH-1228>;

whack: ok, added a require and that helped. Now I see that @codec is nil but I also get "Unknown setting 'codec' for rabbitmq" if I try to set params["codec"] like I do in master

mm

that's odd

mb3: show me an example of what ou want "coordinates" to be set to?

antares_: like LogStash::Input::Rabbitmq.new("codec" => ...) fails?

whack: [-23.234,2323] an array of floats in ES

whack: https://github.com/logstash/logstash/blob/maste... is what I use in master

Title: logstash/lib/logstash/inputs/rabbitmq.rb at master 路 logstash/logstash 路 GitHub (at github.com)

mb3: ahh, you'll probably need two add_fields and a mutate filter to convert it to float

add_field => [ "coordinate", "%{geoip.longitude}" ]

add_field => [ "coordinate", "%{geoip.latitude}" ]

then later, filter { mutate { convert => [ "coordinate", "float" } }

whack: so duplicate fields will automatically be combined into arrays?

should yes

hrm. there should be a mutilate finter.

filter.

interesting, I'll try that

trying to thing what it would do.

jkitchen: use nethack's engraving-fade algorithm on your message?

whack: I don't know those words

:(

ahh

I'm a really horrible nerd, I don't play dwarf fortress or nethack

in nethack you can engrave or write in the dust on the ground

over time the writing fades

it has a clever algorithm to fade it in a way that makes sense in plain text

like 'h' will become 'n' or 'i' etc

Title: Source:NetHack 3.4.3/src/engrave.c - NetHackWiki, the NetHack Wiki (at nethackwiki.com)

"algorithm" is too strong a word, but it's still awesome

maybe it could do like what windows' safemode screen does when you have a broken filesytem

trying to find a good example

whack: is my approach to defaulting to the json codec incorrect?

antares_: no I think it should default to json

current release defaults to assuming json_event (which is just 'json' in master)

whack: how come @codec is nil for me?

whack: http://superuser.com/questions/613702/what-expl...

Title: windows - What explains the garbled message "Start Wandows Ngrmadly" in text mode? - Super User (at superuser.com)

jkitchen: yet another good question marked closed.

what a shithole of a site.

antares_: how are you experiencing @codec being nil?

whack: I was referring to the screenshot :)

jkitchen: yeah I know :(

whack: if I keep https://github.com/logstash/logstash/blob/maste... on 1.1.x, I get an exception about unknown setting "codec"

Title: logstash/lib/logstash/inputs/rabbitmq.rb at master 路 logstash/logstash 路 GitHub (at github.com)

whack: if I remove it, @codec is nil (NoMethodError: undefined method `decode' for nil:NilClass) on the line that has @codec.decode(data)

antares_: are you invoking the rabbtimq plugin via pry/irb?

whack: no, just running the agent

hmm

whack: I can push a branch if you want

that's super weird, @codec should get set when the plugin is initialized

whack: I grep for "codec" and only see it ever mentioned in the rabbit plugin (1.1.x)

I think the logic for special-casing a default codec isn't well done yet

maybe