#logstash

I think grok and convert only support int, float, string

just making sure

new fangled gadgets

is it possible to rename a field name in elasticsearch

damn, I really wish we could figure out why embedded ES doesn't run correctly unless you start LS from within the tarball directory

ronnocol: beta2 should correct that

this smells like a bug to me... anyone care to confirm? http://pastebin.com/duWrnAvt

it's not out yet, though

Title: {:timestamp=>"2014-02-28T18:21:17.815000+0100", :message=>"Trouble sending GELF - Pastebin.com (at pastebin.com)

may as well just set up a separate ES anyway, embedded ES is just for testing anyway

according to the docs short_message should be optional. plus, message isnt!? a required field, afaik

untergeek: I'm running directly off of master as of about 15m ago

really? report that, then

sounds like a jarpath discovery error

I opened a bug

So, now that I have plans for getting logs into ES - It's time to start working on resources to get the data out.. any good dashboards? ;)

untergeek: LOGSTASH-1940

Jira issue [LOGSTASH-1940] LS appears to have issues finding es.jar when started outside of tarball directory - logstash.jira.com - https://logstash.jira.com/browse/LOGSTASH-1940

hmm , deployemnt of not approved code .. does it make sense ?

Coolacid, I'm trying to figure out how to deploy this "Path" on my current setup ... any experience with this ?

Couple ways, you could download the code from the Pull request, or find the patch file, and patch your instance. But, if you don't have experience with this, I'd suggest you wait for it to be completed/ready for inclution in a release.

coolacid : I think, I need to learn to do this then. Comments on github are 4 months old ... it will not be included fast ...

I wouldn't say it won't be included fast.. but I can't say when.

rashidkpc looks to be the one to talk to.. You might also contact ElasticSearch the company and see if a support contract would help with getting it included faster.

last comment is 10 days ago

you could try commenting on it and see whats up.

seems like I need to start account on github :-)

can someone suggest the best SSL method of sending from a logstash host to another logstash host?

Hrothgar-, that's a loaded question -

so it seems :D

If you're purely looking for LS -> LS then you can look at http://logstash.net/docs/1.3.3/inputs/lumberjack

Title: logstash - open source log management (at logstash.net)

doesnt work

But, you might also want to look at a message queue to buffer the messages -- ie: RabbitMQ.

"doesn't work" is a different question altogther ;)

lumberjack, trying it logstash-1680

RabbitMQ gets only 800msg/sec

but it _is_ purely ls->ls

several DCs and want to get stuff centralized :)

Hrothgar-: rabbit can get lots faster than that, turn off durable queues and/or ack

Yes, well, I'm going down that road..

ack is kinda a quirement to ensure delivery tho..

isn't it?

you can get better perf through rabbitmq if you use the compressed_spooler codec

thx, I have tried undurable and ack settings but at best I get 2kmsg/sec

or whatever that name is

odd that tcp input supports SSL, but tcp output doesn't.

oooh whack now you are on to somthing

well "odd"

it looks like I may just send directly to ssl_haproxy->es

Hrothgar-: the other really good thing to check with on rabbit is what kind of exchange you are using

yeah I need HA and no message loss

we have a firehose port that can read 4kmsg + parsing etc

I will check out the commpression idea, thanks folks

hrm... so I am using 1.4.0b1 and just came up against an issue using the geoip filter. Since its not using the flat jar anymore, it can't find the geop files relative to the package, it just looks relative to where I start logstash from :\

try ../../../ :)

addisonj are yo building that jar yourself?

no, 1.4.0b1 does use the flat jar anymore

its just a tar package

I started building my own GeoIP database so I could include internal IP spec/locations ;)

anyone have recommendations of good talks/videos that get into more advanced stuff in kibana (can't quite find a way to do what I want *efficiently*)

coolacid: I am thinking of just bundling up the geoip database into my logstash repo, but I don't quite want to add 18MB of overhead to my source control and deploys :\

it should do like elasitcsearch and have a LOGSTASH_HOME that everything inside the project is resolved relative too

jzawodn I dont know of any, but I too would like to have something to point to for my end users :)

jzawodn - +1, there is surprisingly little documentation on Kabana's more advanced features

mkay. I'll have to try watching what I can find and see if it helps

perhaps I just need to learn ES query language really well and build a UI

Good Evening everyone, I still don't manage to change the timestamp with a date in my log here is an example of entry and my conf : http://pastie.org/private/gsyohdjrwnt67or94qrkq

Title: Private Paste - Pastie (at pastie.org)