#logstash

just need to clean up the mess caused by the code changes, write new tests

more specifically, 1.2 is feature-complete, but lots of bugs

whack: Ok, is it worth for a person who doesn't have any knowledge of internal logstash to jump in and help get 1.2 out the door or should I just chillax? ; P

simmel: totally up to you! I want new folks to be able to hack on stuff if they want to

so if you want to work on something (tests, cleaning up plugins that break now due to recent changes) I'll help however I can :)

I'll have much more time to focus on 1.2 very soon

whack: I'll probably get busy on logstash in a few months anyway (going on parental leave = D)

whack: Not promising anything though, but probably = )

whack: i got a question for you, with the new lumberjack, since everything comes in through the same input to logstash, what's the suggested type => for that?

simmel: all things are appreciated ;)

for 1.1.13 what ES version is recommended? Can we use latest?

professoruss: lemme show you my config

scottbessler: you can use the latest if you use the elasticsearch_http output, yes

scottbessler: 0.26, or latest with es_http

and es_http says its beta, is that a bad idea? :)

scottbessler: beta doesn't mean what you think it means

most modules are beta i'd not be to worried :]

k, thought so

like google beta :)

which is why I've changed the labelling system for that

now it's just milestone numbers

so 'milestone 2' instead of 'beta' - I got tired of explaining what logstash meant by 'beta' ;)

professoruss: https://gist.github.com/jordansissel/a977888d3a...

Title: gist:a977888d3aecbed878e2 (at gist.github.com)

ah, so you use mutate to change it.. hadn't seen that filter yet

so then i can grok the regular types normally after that?

professoruss: yup!

awesome, thanks a bunch

I'm not sure if I understand correctly what grep { match => [ "@tags", "_grokparsefailure" ] negate => true } does...

Can I use this to skip parsed failures to be inserted in elasticsearch?

some days I think Perl is just not the future

(I don't want to insert garbage)

mguezuraga: it should, yes

whack: then I don't get it...I'm testing some very simple like: https://gist.github.com/mguezuraga/6134046

Title: skip failures (at gist.github.com)

but even entering valid data doesn't work...In fact, I have no output with valid or invalid data

indeed, weird

whack: I'm using logstash-1.1.13-flatjar.jar

it's a bug I'd wager

if I run with --debug I see that it cancels events like "hello" and "1.2.3.4" despite one failing grok and one succeeding

ok, so I guess I should open a bug report

hi

Hi

my config reads elasticsearch { embedded => true }

how do i get it to point to the elasticsearch server on the same box

don't set embedded

mguezuraga: yeah, definitely a bug

whack: I'm filling a report now..

<3

okay what do i need to make it them localhost

are there any good doc on the output and how to set it.

yes

the docs

Title: logstash - open source log management (at logstash.net)

Thanks.

when i was starting out i could never find the docs :/

always ended up at the cookbook

eper: hah, oops

Yeah, that is waht i have been looking at.

aye i think it needs making bigger on the website or animated :)

can I just do elasticsearch { bind_host => localhost}

think its host =>

bind_host is for the embedded ES

hmm I am using the cookbooks.

lets try tht and see what happens.

but i recon output { elasticsearh {} } will just work

as it will auto discover it if its on same network - same machine

I will end up with 3 servers. dev stage then prod

so localhost will still work...right?

yeah kinda

you'd need to name them different clusters

if all on same host of course unless host => takes a port number

es ans log stash will all be on the same host.

Oops! Something went terribly wrong.

so you'll have ES+Logstash all on same host but 3 copies of ES for prod/dev/test O_o

:(

Maybe....... 3 server only ES and LS are on each.

New news from newjiraissues: Mariano Guezuraga created LOGSTASH-1240 - Can't ignore _grokparsefailure with grep/match <https://logstash.jira.com/browse/LOGSTASH-1240>;

hey whack, I'm running into an error while trying to make an rpm for lumberjack2-hack with the default "make rpm" command. Do you have time to offer some assistance?

I have this error in the logs

{:message=>"Failed to index an event, will retry", :exception=>org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [1m

sure ES is running ?

looks like it

0027 10876 1 0 13:06 ? 00:00:15 /usr/lib/jvm/default-java/bin/java -server -Djava.net.preferIPv4Stack=true -Des.config=/usr/local/etc/elasticsearch/elasticsearch.yml -Xms2048m -Xmx2048m -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/usr/local/var/run/elasticsearch/dev_05_dev.pid -Des.path.home=/usr

/local/elasticsearch -cp :/usr/local/elasticsearch/lib/*:/usr/local/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.ElasticSearch

tps://gist.github.com/MixMuffins/6134263 <- that's the gist for the error I'm getting; not sure why it's acting up, other than perhaps a different version of ruby or go; I did this successfully on my host machine with no problems. :/ any idea what it means by unknown file type?

https://gist.github.com/MixMuffins/6134263 <- that's the gist for the error I'm getting; not sure why it's acting up, other than perhaps a different version of ruby or go; I did this successfully on my host machine with no problems. :/ any idea what it means by unknown file type?

Title: gist:6134263 (at gist.github.com)

version 0.20.6. of elastic search ?

I dont know.

umm. let me see.