#logstash

yes

i tried configuration with two outputs - one to elasticsearch and one to stdout

sodar: maybe the other entries are filtered separately by syslog and not sent to LS

default syslog setups on most distros do that

cassianoleal: is has been nice with me so far, any probs for you?

rpetre: it's worth to check out. few days ago this setup was working, so maybe something changed in syslog setup

jpodeszwik: Performance is unchanged, maybe if anything it's worse.

jpodeszwik: ok, .ww works.

vangap: no, I haven't upgraded it yet. I'm still studying the upgrade process. :) I have had problems in the past when the ES version differed from the one bundled in LS, but at the time I was using the node protocol. Just checking, really. Thanks!

rpetre: it is set to forward everything to logstash

Yes, I actuall I also ran into that issue, but I didn't know about http protocol initially , You can replace the ES directory in logstash at vendor/jar/

but http protocol seems cleaner option ^^ cassianoleal

and with two outputs it was printing those missing lines to stdout, but not to ES

vangap: yep, I changed to http when the issues started appearing and never looked back

vangap: it's good to know that I can just replace the ES jar in LS if I need to, though :)

jpodeszwik: "Tip: there's an optimization in the way of the Use event times to create index names option. Since Logstash creates an index every day, Kibana uses that fact to only search indices that could possibly contain data in your selected time range." Does this have any effect if the index pattern is changed to weekly/monthly

oh, another question -- does Kibana 3 work with ES 1.4.4? I'll be upgrading Kibana as well in time, but I don't want the users to be cut off from the logs during the migration

cassianoleal: yes, it works

rpetre: brilliant! thanks!

if you're upgrading from pre 1.4 versions of es, you'll probably have to set something connected with cors

it just makes your browser do http calls in ES

those didn't change much

cool

vangap: i guess it does

jpodeszwik: hmm, seems like I need too look into that once then

vangap: i have one monthly index and kibana handles it fine

is there any general thing about the no of docs we can keep per index? performance wise

jpodeszwik: ^^

I have around 300-400k events per day atm

Way more than that.

i got 150M per day in my biggest indices and its fine

i guess it depends on hardware configuration of your cluster

jpodeszwik: You guys ever performance tested logstash on kafka then? I'm getting around 7k/s throughput on fairly small json messages.

Trying to figure out if that's normal, or way off the mark.

Poooogles: what hardware configuration do you have? is logstash using all your cpu while working?

~80%, quad core box.

Haswell Xeon 2.3's.

Where can I download Logstash Forwarder CentOS x86_64 packages?

Only unusual thing I can think of is I'm using unicast.

Poooogles: im filtering most of my events, so im not sure how how much can i afford, didnt performance test that

Poooogles did you use logstash without kafka input and was it faster?

Hi everyone

jpodeszwik: We've got a firehose of 150k qps, so using it without a kafka input isn't really an option. I guess maybe I can pull a file of a few million lines and test against that.

Poooogles is that 80% of one core or all cores?

All.

60 -> 90% usage over the past 10 minutes or so.

Poooogles: jpodeszwik we haven't done any perf testing yet on the kafka input/output.. we are planning to build something to do perf testing for the plugins.

electrical: Mmm right ok.

Shout if you want any help testing as I've got data coming out of my ears.

Poooogles: feel free to open a github issue about your performance results so we can have a better look at it internally and also talk to the ruby lib developer of it.

I'll submit one over the weekend.

hello people

need some help

i have a nginx-access log and it doesnt parse with NGINXACCESS

any clue

hi, I have a field 'request' I want to remove 'amp;' (leaving the &, before using a kv split) from the value (as manytimes as it appears) - which filter/method should I be using?