#logstash

hello

is it possible for logstash to not load default grok filters?

matejz: I don't think so. What problem are you trying to solve?

torrancew: I think I could get it to start faster with default patterns removed

I'm not sure that the payoff there would be worth the time required to do it

A lot of your LS startup time is in the JVM and JRuby warm-up

torrancew: it takes 30s+ to load now… Would love for logstash to start faster:)

JRuby in particular has this pain point

oo ok

The JRuby lead suggested recently that he's close to a breakthrough here, via twitter, IIRC

that’s good to hear:)

There's a JRuby page on speeding up startups

most of the time startup is not a problem, but when developing new filters and running pipeline tests, I would really love for it to start a bit faster:)

but most of it benefits development envs more than prod envs

(as it does things like disable/reduce the JIT, which leads to quicker startups, but slower overall performance)

Thanks for the info, I’ll have a look

I need it for dev anyways, so performance is not such a problem as I’m not running a bunch of logs againts it:)

one more thing… do you know of any repo / page with logstash configurations from productions?

Example configurations are easy to find, but actual bigger configuration used in production envs are hard to spot / non-existant

matejz: What are you hoping to learn from such a repo?

I think the main reason they're hard to find is that most of the time it's littered with business logic / data that may be harder to get approved to share openly

torrancew: just how people are doing it, how their pipelines look like and to maybe get some new ideas:)

yea, probably

Yeha I don't know of any sort of at-scale examples

coolacid has a repo called GettingStartedWithELK

but it's mostly specific cookbook type stuff

I see…

The smaller examples will show you how to do things. Asking here may be your best be for 'should' and 'why'.

Any large pipeline won't be using your structure.

I’m not sure if my pipeline is considered small or large… I’m probably somewhere in the middle

with around 30 different services sending logs to LS

and flow rate around 1200 logs/s

but only around 10 servers out of 100 sending logs

/nod I'd say that's on the high end for source types, and middlin for volume.

Why I was asking for examples is right not my pipeline is fast enought, but I have a feeling I could squeeze a bit more out of it:)

one think that I’m asking myself is if I should look towards generating JSON logs on service side or just GROK human readable logs

one example is apache

then there is syslog and so on

Moving the structure closer to the source will reduce load on your logstash, so ingesting json logs would help.

If, however, you don't have tight control of the sources, then that's not so easy.

For example, we accept sources from all over campus. I don't want everyone to be able to name their own fields.

Hi

I have a question about dead_letter_queue plugin. is this the correct place to ask?

it's related to https://discuss.elastic.co/t/error-in-processin...

my DLQ plugin is generating a single byte file which causes parsing in the pipeline which consumes the dlq data