#logstash

is that something from previous versions which have now been refactored into the popup you just described?

I'm guessing that is either from an older version

or some concepts rashidkpc was playing around with

ok. thank you. you have been really helpful :)

If you want

check out Kibana 3

Title: elasticsearch/kibana3 · GitHub (at github.com)

and there's a demo of it at demo.kibana.org

thanks

logstash => es => kibana feels also a lot faster than the previous logstash => graylog2 => es which I tried

the graylog2 pipeline kept the machine loads around 12 (8 cpu machine) and the current pipeline barely brings the load up to four (and it's closer to peak hour than what it was when I tried graylog2)

Garo_: thanks for the info

in both setups, all components were on a single machine?

yes

which version of graylog?

latests.

cool, thanks

actually, there was a change in the underlying disk system

so that pretty much voids the results

sorry :(

what was the change

I have four EBS disks. previously those were combined incorrectly with lvm so that the lvm did not do stripping

and now they are properly stripped so that the io will be divided equally to all disks

does logstash-flatjar.jar command line support "--pid PIDFILE" parameter?

jkoppe1: you filed the issue about that 'absent' still created some files right? ( puppet-logstash )

jameshyde: http://logstash.net/docs/1.1.10/flags

Title: logstash - open source log management (at logstash.net)

hey, anyone ever see this before when starting logstash?

The signal INT is in use by the JVM and will not work correctly on this platform

thanks devOpsEv. unfortunately --pid is not available.

java 1.7.0 on RHEL 6.4

Electrical: yessir

devOpsEv: I added redis back in the mix and now my @type is coming through correctly. Thank you sir

no problem :)

I still don't know what to use as a shipper though

logstash

lumberjack doesn't do redis output

logstash -> redis -> logdstash -> Elasticsearch -> Kibana

I can't use logstash. Don't have java on my prod servers

sudoSamurai: well that is a problem I guess, there a reason you can't put java on them?

anyone peeked at this yet?

Title: Edis (at inaka.github.io)

Edis is a protocol-compatible Server replacement for http://redis.io/, written in Erlang. Edis's goal is to be a drop-in replacement for Redis when persistence is more important than holding the dataset in-memory.

devOpsEv: yah... security guys can't distinguish between jvm and java browser plugin, so we are not allowed to put a jvm on the prod servers

are you f-ing kidding me? lol, sorry, that's the most ridiculous thing I have ever heard

devOpsEv: if it has "java" in the name, it is bad joojoo

and they let you use FOSS?

devOpsEv: yep. Well, even if I could use it, logstash on the shippers runs a little heavy anyway

well LS has plenty of inputs

Finance -> Security -> me. My security guys are fussy until you tell them the commercial replacement for logstash (splunk in this case) is almost $225k. They seems to STFU then.

splunk, ugh

devOpsEv: yeah, I'm debating on using beaver. Lightweight, and I can use python

used the evaluation version for a week, it was terrible

just haven't tested it out yet

nah, I really like splunk

it's just stupid expensive

just couldn't get it to do the things I wanted it to do

it would be expensive if it were half the cost

I just can't believe they won't let you run java at all

that's too funny

not on public facing machines

I don't know what kind of business you're in, but do they realize that like, banks use java on the backend to run public facing webapps?

I can't say I blame them. The less you have on machines that are on the Internet the better. Installing java on production boxes just for a log shipper is a little unreasonable

I suppose you have a point, but most OSs come with java preinstalled, Java is practically a requirement for everything these days

limit attack vectors, etc...

not that coming with it preinstalled is a reason to use it

jkoppe1: issue should be solved now. can you test it out?

but you get what I mean

I mean, shit, Android apps are all Java and the entire network of Android phones hasn't been hacked/compromised yet

I understand, but it's a battle I don't really want to fight considering how easy it is to work around it

haha

yeah. just shooting the shit, not trying to reason it to you or to your sec team

:)

trust me, I agree with you. I'm a former java dev.

I'm just a lazy former java dev, and when it comes to intra-company politics, I want nothing to do with it. ;)

I hear you there

now, back to banging on grok...

howdy folks!

hello

hey whack

whack: hey, can I run something by you that I found?

hiya whack

sup?

nothing much. and there?

did some early house work this morning. spent 3 hours in the attic

then it got too hot :(

hi whack :)

Electrical: yea, i'll check it at some point this week. today's not a good day

whack: so I'm using logstash as a shipper and indexer using tcp as the in/out. On the shipper side, I specify the type per file. On the indexer side, I have to specify a type as a requirement. That indexer side @type is overriding the previous type

jkoppe1: that's fine.

sudoSamurai: you'll want to set 'format => json_event' on your tcp input

I did

didn't matter

is it format, or message_format?

format.

oh. my bad.

I can go back and test, but I'm pretty sure I tried that

I tried message_format as well

sorry sudoSamurai, I was so close

I got around it by using redis. :)

<http://goo.gl/Vqldu>; (at gist.github.com)