#logstash

hey all.. for some odd reason when i launch the logstash web, instead of seeing a logstash theme I get a default kibana theme

<http://tinyurl.com/ka9u6fa>; (at logstash.jokefire.com:9292)

this was the command I used java -jar logstash-1.2.2-flatjar.jar web

from the logstash directory.. any idearrrrrs? tx

can someone point me at an article that compares logstash and fluentd?

(I'm sure everyone here will say logstash is better, but I'd like to understand all the reasons why, ya know)

evening

jumpkick: i haven't used fluentd, unfortunately :)

i don't know of any comparative analysis

they have some interesting plugins

alveen: k, it's weird because I originally found out about logstash when I saw a google hit on fluentd + elastic + kibana

ahh

ok

though it seems logstash + elastic + kibana is the more "normal" fit

yeah

i mean, given that they're all three run by the same company now :)

yeah

s/run/developed

There's a bunch of stuff I need to figure out about the stack, like security and high availability

it's interesting that fluentd is written in ruby

looks like there's some client/server SSL stuff in logstash now (or maybe it was lumberjack)

avleen: as is logstash?

i can see that being nice for many people. logstash is runy too, just wrappy in jruby for preformance

*performance

yeah lumberjack ships logs over ssl

I'm thinking a lumberjack -> some kind of MQ -> logstash -> elastic -> kibana would be the best stack

for HA

but securing everything could prove painful

Redis now does a degree of HA

saw this one: http://logstash.net/docs/1.2.2/tutorials/gettin...

<http://tinyurl.com/nddxncg>; (at logstash.net)

the design of fluentd (looking at the source code) is actually not too unlike logstash.. logstash has 4x more forks, and 7.5x more commits. logstash's first commit to github was august 2009. fluentd was june 2011

jumpkick: well, "HA".. if you have multiple logstash instances, you don't need MQ

lumberjack will rotate between the live ones

does it do some kind of load balancing or does it just pound one till it fails and then move to the next one?

pretty much that. I'm going to add in some load balancing logic this week

the splunk forwarder rotates connections every so often, so I'll probably do similar. every <timeout> seconds, reconnect to a new random server

avleen: some way of distributing load across the catching servers would probably be the best, not an easy problem to solve

jumpkick: if all of the clients randomly reconnect, the load would balance out well :)

too much reconnecting (every few seconds) will also be bad for performance

naa

yeah

every 60 seconds, would be quite OK

I suppose

not like, every second :D

but one a minute, or so, it would be quite OK. it's very fast to reconnect

yeah that'd work

from lumberjack github - "Redis development refuses to accept encryption support, would likely reject compression as well." :(

yup :(

you could write a tcp proxy that supports encryption in node.js

or use haproxy?

i've generally found tho, that as long as your logstash downtime is "short" - less than the time before your logs rotate - then things usually recover automatically quickly. I found (at our scale, anyway), adding redis / MQ made performance significaly worse

micko: or write a codec which endrypts the data

*encrypts

having recent written a codec and a filter, i have to say that they are super easy to write

avleen: got anything on github you could share?

lumberjack directly to a logstash cluster sounds like the best way to go; simpler and more secure too

I think I read it does client certs, client ssl keys or something, which is good

keep intruders to the network from flooding the logstash with spam

micko: sure!

or at least they'd have to break into a node and then they could be filtered out

avleen: thanks :)

micko: https://github.com/avleen/logstash/commit/47584...

<http://tinyurl.com/p46dlgf>; (at github.com)

i sent whack a pull request

it collect multiple events, compresses them, and then sends them to the output

cool. i'll have a look now

this was while I was trying to improve the performance with MQ (which has an uppser limit of 40k events/sec)

with the compress_spooler codec, I was able to get.. well, a lot more

I maxed my input at 160k/sec, and MQ was still fine

nice

it was really cool for shipping data from one logstash to another via zeromq too, if you do that.

lol... that's pretty good 4x faster

in the end i went back to just having one layer of logstash servers

but I know others would benefit

zeromq better than redis for queueing?

eh... depends

i don't full understand zeromq, so i don't know how it buffers, how to make the buffer persistent to disk, etc

avleen: re: fluentd; fluentd was heavily derived in design from logstash; I couldn't get a solid answer about why it exists though.

avleen: zeromq doesn't persist to disk

avleen: ill have a play with both

whack: Redis can now :)

micko: redis has been able to persist to disk for a while

my bad

hehe

im getting confused with it's new HA I think

whack: yeah, that's what I struggled with too... why the heck did they bother to write fluentd (also written in ruby, also using plugins for in/output, does the same thing from what I read)

whack: yeah that's what i was seeing too (re: fluentd). eh!

sometimes people want to write code, to write code.

^^^^^

personally, i try not to discourage creativity. sometimes you just have to reinvent something to satisfy a need in yourself

and that's ok

I don't know why they didn't just fork though

can I quote that :P

hell, i've done it enough times myself

Hello! Whenever I start up logstash with the Twitter input, I get "A plugin had an unrecoverable error. Will restart this plugin." Can anyone here help?

micko: sure :)

micko: Attribution: "Avleen Vig"

:-)

lol

whack: how's the newborn? do we have a name yet? :-)

avleen: indeed; when I asked them why, I only got "Because we only want to support json, nothing else"

maybe there was just a preference for apache config file syntax (yuck)

avleen: which is fun because their plugins support more than json now

some people gonna code and not contribute, it happens.