#logstash

hi there, completely new to this, have setup ELK on Centos 7

hi guys! we're running bro on machine and we want to forward logs from this instance to the ELK stack running on a different machine.. i guess we have to set up logstash forwarder on the bro machine?

anyone knows the format to put several logs (from client servers that is) on the logstash config file at server ?

zamba: yes

puttylla: do i do any parsing on the logstash forwarder, or is all that handled by the logstash server?

server does afaik

ok, cool

bodik: we fixed that in 1.4.3

and can someone please explain the concept about indices?

zamba: be aware I just setup this first TODAY

puttylla: with bro?

zamba: that is a way to structure data and tag logs for allow some speed when searching otherwise it would be very slow

zamba: without, but same applies

ELK

puttylla: well.. how many indices should one have, then?

whack: fixed ?> i see different versions in repo for 1.4.3 too

whack: logstash: 1.4.3-1-7e387fb logstash-contrib: 1.4.3-1-bf2ee84

whack: the reason oi'm fighting it that in 1.4.3 mongodb output is broken ;( (i know it's contrib ...)

what's a recommended java library for logging? we're currently using slf4j

so i wanted to fix version to 1.4.2 which works, but cann;t do it through puppet module ;(

bodik: uhg :(

oh, and log4j 1.4.2

bodik: those version numbers are fine, what's the problem?

howdy

bodik: the "Version" is 1.4.3, the "release" is 1-7e387fb for logstash, 1-bf2ee84 for logstash-contrib

the release has little impact package

whack: the problem is that class { logstash: repo_version=>"1.4", version=>"1.4.2-1-xyz" }

it would install logstash, but not logstash-contrib

because build version numbers are not in sync for logstash and logstash-contrib packages

I'm curious if anyone has ever had problems with LSF not reading other files explicitly defined in its configuration? Granted, the first file in it's config is the largest (2GB/hr) it never gets around to some of the smaller files.

bodik: don't' set version that way

bodik: set version 1.4.2 or 1.4.3

not 1.4.2-1-xyz

quick one, anyone knows if you can use a path with wildcarding ? say I have 100 chrooted apache vhosts with each log in the custom folder

whack: i tried that too

so I do not want to put path for each of the files (as it is impossible to mantain)

whack: same story, apt get install logstash=1.4.2 >> version does not exist

i'll check that again

whack: http://pastebin.com/n9CgcRQU

Title: err: /Stage[main]/Logstash::Package/Logstash::Package::Install[logstash]/Packa - Pastebin.com (at pastebin.com)

bodik: maybe this is a bad assumption in the puppet module

I have a problem with kibana being unable to fetch mapping

the release numbers should not need to be the same

the version numbers must match

bodik: try version => "1.4.2-*"

I have no default index patern, but do not know what is should match

whack: it looks that i use old version of puppet module for logstash too

whack: version => "1.4.2-*" works

whack: thank you very much .. ;)

woot

afternoon!

avleen: avleen !

WHACK!!

how're you doing man?

hanging in there.

well, that's about all we can all do :)

finally managed to upgrade to ES 1.5.2 last night, so nwo I can start rolling out Kibana 4 :D

wooot woot

avleen: 1.6 will help you also

wrt fast restarts

i saw that! oooooooooooo i'm so looking forward to that!

I don't think i told you this, it's been a while since we spoke, but I hacked up the date filter in an interesting way

we were getting a lot of logs with wild timestamps (mostly because clocks on things like mobile clients are completely untrustable)

and it was blowing up the number of indices we have

how'd you solve it?

I am confused, if I use rsyslog I do not need logstash forwarder necessairly right ?

puttylla: you can use rsyslog instead of logstash forwarder, yes

so I added to the date filter, to only allow events where the timestamp is +/- 1 day from today, and to log all the rest so we can parse them later if needed

whack: so if I sent the logs I just have to parse with elasticsearch, logstash server itself does not do anything.

?

avleen: mobile client == a web browser? or etsy's app?

etsy's app

avleen: any desire to have time within the app wrangeld with ntp? :P

in-app ntp hehe

which can do things like bundle log events while the user has no reception, etc

makes sense

welllllllllllll funny you should mention that!

i've raised the same idea :D

it should totally be possible to fake it, right?

poll a URL for the current time, calculate an offset from the phone's clock, boom.

whack: disregard my comment, obviously logstash server will parse data that comes from rsyslog

whack: we also started deploying logstash 1.5 too. got it on one or two smaller clusters right now I think, and going to push to the main cluster in a few weeks.

avleen: right? At least getting close anyway

totally

avleen: still, +1 on the workaround

we do what we can because we must

hehe

:)

also I saw this, this morning:

<http://tinyurl.com/noxoo2w>; (at stories.iovation.com)

so is the problem still that an offline device may show up days later with good-timestamped-logs but will cause your frozen indices (2+ days old) to receive new documents?

yeah that problem would still exist. and I'm wondering if I might do better with sending those to a separate index. The volume is low enough that I could have month-based indices with day-based alises

and kibana can query multiple indices at once, so that would be ok

yeah

certainly an interesting problem :)

maybe having a 'stray/old' logs index that you compact weekly or something

yup!

if older than 1 day, send to the stray-old-app-logsYYYYMMdd thing

then you don't waste your daily _optimize calls

yeahhhhhh! And give that index a bunch of aliases

with YYYYMMDD

+1 there

omg this is genius

whack++

avleen: we had a 1B document day a while back. The cluster didn't handle it :(

rastro_: I'm up to 3bn/day. We were much higher and then cut back some unnecessary logging. but we'll be at ~6bn by the end of the year I suspect.

I think curator can even maintain your index aliases for this also

but not sure

i *think* it might be able to too, i know it does something with aliases.

curator does do aliases

rolling, particularly

right now I have a hacky python script which maintains some common aliases (logstash-1d, -2d, -3d... -1w, -2w, etc) to point to the right recent indices

lol!

I am magic, or at least IRCCloud's notifications make me so

avleen: Curator will do those for you

i picture a command center with a klaxon.

untergeek: brb throwing my hacky python away :)

or you could use the Curator API to help, if you wanted to keep using your hacky python

dude i never want to use my hacky python, even when i just wrote it and i'm proud of it.

:)

literally the moment it's done, I get that "urgh, I don't want to do this" feeling :)

avleen: https://www.elastic.co/guide/en/elasticsearch/c...

<http://tinyurl.com/q9e7324>; (at www.elastic.co)

thank you!

woooo

man i've missed chatting in here the last couple months you guys are amazing.

avleen: <3