#logstash

Sure, one sec

hello all

can some tell me how one sets the HEAP size for logstash

ghanima: LS_HEAP or something like that IIRC

restro: Here is part of the text. arg="" realm="NX:Secure"

Lots of stuff like that in the message

torrancew: so I am running this inside my shell

export LS_HEAP='20g'

ghanima: LS_HEAP_SIZE in /etc/sysconfig/logstash for me.

ghanima: how do you start logstash?

bin/logstash -f configfile

by hand?

rastro: here is the pattern: gsub => [ "message" , "\"" , "Null " ]

oh sorry

torrancew: yes I am testing something locally right now

LS_HEAP_SIZE as rastro said

Simon_k: are you using the kv filter?

NOT LS_HEAP

rastro: yes

Simon_k: did you try trim there?

I am gsubing than using kv. otherwise kv gets wrong fields

torrancew: so in my case this is still not getting set correct when looking at the -Xmx setting

I am seeing this -Xmx1g

rastro: Thank you so much. This did it. I own you a beer :-)

can you possibly share some actual pastes?

we have no real way to know if you're doing this "correctly" (though I dislike that term)

sure, one min

Simon_k: gsub doesn't do quotes: https://github.com/logstash-plugins/logstash-fi...

<http://tinyurl.com/z867zh9>; (at github.com)

xeorex: glad it worked.

Simon_k: maybe we can help you gey kv working. it's usually much nicer than the alternatives.

Here is the log I am working on. I figured I would replace the quotes with null, than KV would be able to handle it.

id=firewall time="2016-06-28 15:30:01" pri=6 fw=x.x.x.x vpn=ive user=Nx\xxxx realm="Nx:Secure" roles="Nx" proto= src=x.xxx.x.xx dst= dstname= type=vpn op= arg="" result= sent= rcvd= agent="" duration= msg="xxxxxxxxxxxxxxx "

But the log differs in length and sometimes feilds are empty with just a blank space and sometimes just has an empty quotes.

torrancew: is the me setting the variable and executing logstash http://pastebin.com/tgWRMFgE

Title: [centos@ip-10-1-1-180 logstash-5.0.0-alpha3]$ export LS_HEAP='20g' [centos@ip-1 - Pastebin.com (at pastebin.com)

But obviously many times those fields are not empty and they hold valuable info

Simon_k: ah, you'll probably get caught by this one, too: https://github.com/logstash-plugins/logstash-fi...

<http://tinyurl.com/jovra9r>; (at github.com)

torrancew: this is full java args when that process runs http://pastebin.com/NxA1qyzu

Title: Full java process /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: - Pastebin.com (at pastebin.com)

ah, 5.0alpha?

rastro: sop far that has not affected me but its possible

could e a bug or that the flag has changed

Simon_k: in your example, you had msg="xxxxxxxx " which would technically fail.

ghanima: try setting the JVM opts you want directly via "LS_JAVA_OPTS"

ghanima: it looks like this is changing come 5.0

Simon_k: so i take back my recommendation for kv{} :(

rastro: Ah I see, so then my non-empty quotes will fail as well.

Simon_k: "foo" is fine. "foo bar" will break.

Ah got it.

Simon_k: and since gsub is broken, i think that leave you ruby{}.

Then I am not really sure how to parse this log. Especially since there are fields which are not in every log, and the log length differes.

That is one issue I have not been able to figure out in Logstash, how to treat logs that are not always the same.

Simon_k: kv{} is normally a good solution for that, if you have a good separator.

Simon_k: e.g. urls with "&"

I will look into this. can kv{} only be performed if there is an = sign present?

torrancew: sorry had to reconnect did you have a chance to look at my pastebins

Simon_k: you can change the separator with value_split.

awesome. I really appreciate your help. I am still wrapping my head around the parsing. I will look further into this.

Simon_k: sorry there wasn't better news. too many open bugs.

yeah, looking forward to an update :)

I'm having difficulties using winlogbeat via logstash.

I keep running into these two errors:

ERR Failed to publish events caused by: EOF

ERR Failed to publish events caused by: read tcp

ghanima: I looked at the 5.0 stuff and it seems the LS_HEAP_SIZE setting went away. Use LS_JAVA_OPTS and set your jvm flags directly.

torrancew: thank you sir

good luck

please report back if it works

(or doesn't

hey all - we are getting random failures connecting to logstash: 2016-06-28T20:35:16Z ERR SSL client failed to connect with: dial tcp X.X.X.X:5044: i/o timeout

it's happening on all of our filebeat procs... and its completely random

any idea on where to start debugging something like this?

sarkis: telnet?

not gonna help much with ssl. s_client maybe

the issue is that its intermittent

io timeout sounds more like a network connect error

but often that points to some part of the pipeline taking too long

i just noticed we are using logstash-input-beats 2.0.3

filebeat waits for LS to ack each batch, unless the timeout fails. Sounds like the timeout is failing

i should probably upgrade the plugin too

never a bad start.

https://golang.org/src/net/net.go#468 line 468 :)

Title: src/net/net.go - The Go Programming Language (at golang.org)

where is the default patterns dir for logstash if I installed from a .dep?

mspo: sure, though that's not particularly useful data

torrancew: neither is the error message

Not contesting that

Is there a way to parse tags and use them as vars? ie. path => “logpath/%{TAG1}/${TAG2}/etc.log

is logstash jumping version numbers?

iamchrisf: "tags" is an array. No clever tricks available on it, what you describe is more like a "normal" field

mspo: clarify?

from 2.x to 5.0?

ah, yeah, all ELK products are going to converge ona shared version

and kibana already went up to 4

so they all had to go to 5

torrancew: yea i was wondering how I can reference the individual elements in the array

you can index in, I think

but nothing fancier than that

[tags][0] maybe

"%{[tags][0]}"

torrancew: yea I think that will work.

Going to give it a shot. thanks

good luck!

ah ok u guys are right looking at hte logs...

{:timestamp=>"2016-06-28T20:58:18.304000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}

Anyone know why I'm getting EOF errors and read tcp i/o timeout errors?

using winlogbeat?