#logstash

jsvd: sorry, got disconnected.

Thanks for the tip

Hi, all. I asked a question in #elasticsearch but figured it was best to do it here. Hi all, I am upgrading my logstash from 5.6.9 to 6.2.4. For all that is filebeat, everything is fine. For all that is logstash. I get this king of error : https://pastebin.com/bfb5SEhS. I know it has to do with the removal of type in 6.x. I'm just not sure how to go at it.

You need to change your filters to only send a single type value to elastic. A mutate right before the output stanza works. If you use type elsewhere (filters, queries, reports, etc.), you'll want to preserve the original type value in another field.

If this help, here is how I have my input and output. https://pastebin.com/iqCqB0gr I can add the filter but there is nothing special beside some grok.

If you query your index, what type does it show on the documents?

right now in 5.6.9 _type and type says cloudfront For that index named logstash-2018.05.29-cloudfront

You might want to set document_type. It will default to 'doc'.

I will go read into that !

Hooo I see

I think it is all because I applied some logic under logstash declaring type for logstash document to apply the "correct" filter and then output to the cluster.

If I understand correctly now, which I did not a few minutes ago. The field type will be remove completely under 7.x. So If I stop using them right now. it could fix current and future probleme.

Future problems. Right now it still has a default value, which doesn't match your data.

Noted. I will add the document_type to my config and upgrade back to 6.2.4 and see how it goes.

Good luck.

darkmoonvt, Nice it does seem to fix it! I really misunderstood that part of the upgrade process

darkmoonvt, Big thanks for your time and help! Now it is working. Would you still say that I should look into that usage of the type field to filter under logstash and try to "fix" that now in preparation for 7.x?

darkmoonvt, Reading again through the doc. If I was using the new multiple-pipeline from 6.0 release. I could have each log type go through its own pipeline and so I would not have to use if type to apply the correct filter.

Or can they only have their own .conf file per log type and have the same result. Ill need to read more !

or maybe simply using tags instead of type and be future proof too.

Yes, that should work. It will depend on the rest of your environment.

(multiple input ports for different 'types' isn't an option in my case, for example.)

I have a beat input which kind of take care of itself. Same for my S3 input, though I did apply a type to filter out. Could be a tag or a separeted pipeline. I have 2 udp input using different port to add the proper type. You say it is not an option in your case. Can I ask how you separate your log?

Almost all of my input is via *beats. We don't control the sources, but can require a few custom fields. We started out using the type/document_type setting in beats before we learned that the elastic type was being removed.

Now, we're moving to fields.type, and we route to the right parser based on that, mostly.

(syslog, http, snmp, netflow, argus, bro, duo, o365, etc.)

Noted, that is also what I am doing under beat right now. So I will keep that in mind ! Thanks a lot again for your time and for sharing!

Glad to help.

I need some help with my logstash. Is someone available to work with me?

Can someone tell me how to right size my heap space? I have an error about that with my flows.

What's the error?

[2018-05-29T00:04:34,450][ERROR][org.logstash.Logstash ] java.lang.OutOfMemoryError: Java heap space

[2018-05-29T09:35:46,710][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"ppsat", "exception"=>"Java heap space", "backtrace"=>[], :thread=>"#<Thread:0x2a82ccd@/usr/share/logstash/logstash-core/lib/logstash/pipelin

e.rb:247 sleep>"}

Try increasing it gradually. For logstash, increase -Xmx but you can leave -Xms alone.

What about elasticsearch?

What about it?

should I increase it as well?

Not if Logstash is the only process complaining

I have 5 pipelines configured and only 3 are creating indexes and none are updating.

Why not?

I don't know

Does this mean it put the pipeline to sleep? [2018-05-29T11:57:41,183][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"elastiflow", :thread=>"#<Thread:0x8623164@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}

It's probably waiting for input

I can confirm with Tshark that it is getting data on the port it is supposed to be watching on

There are only 3 instances of java starting. Is there some magic limit to the number of workers that can be started?

Wouldn't think so, but I haven't worked much with separate pipelines.

Increasing the memory has 2 of the pipelines updating. Should I add more to get the others to start??

Yeah

Ok giving it a shot lol

Dang still only start 3 java instances.