#logstash

Is there a verbose flag when running logstash-forwarder by hand?

cully: exactly!

hi... i'm having a really hard time trying to figure out how @timestamp gets set.

i'm using a syslog input and i'm relaying from my central rsyslog server with the RSYSLOG_ForwardFormat macro

pieterl: by deafult, ls assigns the time it received the event to @timestamp

but you can use the date filter to adjust it

ah, right.. but then it sets the wrong time :)

do NOT adjust it by hand, as it's not a string, it's a ruby Time object (or Date, or something like that)

how so?

i'm UTC+2 (Europe/Amsterdam) and it's showing Z at the end

yes, LS *STORES* all times in UTC

but at the UI layer, you can *DISPLAY* however you'd like

it's done for normalization

yeah i get that :) but it's logging events from now "in the UTC future" as it were

what exactly is the cause of trouble in your case?

2014-07-30T23:00:37+02:00 is in my central rsyslog "to file" storage, so rsyslog is receiving the correct time

mhm

but when i look at logstash (rubydebug) 2014-07-30T23:00:37Z would get logged in @timestamp

for the same event/entry

ah, I see, so it's not actually converting to the RIGHT UTC time?

there's only one UTC time as i understand it :p

and for me, logstash is 2 hours off hehe

what I mean is, that the timestamp it's generating, is not the actual UTC time of your event...

yes

so that's fixable

can you share your config?

at least the relevant sectiosn here

yeah lemme copy paste some gists

hold on.. and thanks :)

specifically, where you capture the rsyslog time (probably in grok), and the date filter that uses it

np

sorry for the miscommunication

doign like a billion things at once :)

arent we all :)

what'd you need except for the logstash.conf and rsyslog.conf?

<http://tinyurl.com/num9rde>; (at gist.github.com)

and, if i needed to grok that data first, that's fine.. :) i just figured i'd try a very simple setup first

there's some logs that cannot be parsed (grokerror tag) and they do receive the correct timestamp

pieterl: do you not have a filter section?

no

this is it for logstash itself

pieterl: I'm not following

there is nothing else configured regarding logstash

ah, I see

pieterl: you've discovered one of the many short-comings of my least favorite input - syslog

ohw :(

pieterl: let me give you my own, unprofessional, biased opinion on the syslog input

it's a meta-filter

syslog is sugar, around tcp/udp inputs, combined with implicit grok and date filters

and it's the source of all pain in the universe, I suspect ;)

mostly because no 2 vendors agree on what "sending syslog protocol" actually means

i think the only thing everyone agrees on is "ascii on port 514"

Sketch: yup

true.. but rsyslog is quite versatile and.. would seem to be accommodating logstash here

pieterl: the problem isn't rsyslog. its' that no matter what logstash does, it won't match every syslog ever. It's probably a "bug" in the syslog input, but fixing that bug probably breaks some *other* syslog shipper

I would highly recommend to anyone using the syslog input, that you consider rolling your own - tcp/udp, syslog_pri, grok and date should be all you really need

damn. :)

and in that way, you're free to adjust and work around the input's shortcomings

syslog_pri?

at one point, syslog was even deprecated as an input

it's afilter

that takes the <xx> priority, and parses it into actual fields

ah

it's what creates facility/facility-label, and the severity analogs

I bet GSwithELK has an example for that

so.. any way i can monkey patch around this? rewrite @timestamp?

pieterl: short of my suggestion, not really

the syslog input is throwing away your tz info

though...

pieterl: is the timezone on that server also +02:00?

yeah every server is +02:00

Europe/Amsterdam in /etc/localtime

ok, then in that case, yes, you can

add a filter section, with a date filter

use "timestamp" (not @timestamp) as the match

what that'll do, is go back to what syslog captured for you. you'll need to specify the right pattern for the filter, but if there's no tz, date will assume to use the server's local time

what if any the perfered choice udp or TCP as an input?

beata: it's somewhat philosophical/depends on your needs

I use TCP-based inputs

performance is a main factor im looking at a high ingest rate.

no real difference AFAIK

tcp overhead is pretty minimal on a modern box

got ya

Anybody have any experience with logstash-forwarder?

I'm getting: Failed to tls handshake with 172.17.0.12:5043 tls: either ServerName or InsecureSkipVerify must be specified in the tls.Config

I can hit that endpoint with curl -k

clarkfischer: it's the -k bit that's the problem :)_

cert needs to be valid for the name/ip you're calling it by

it was more permissive in the past, but a golang change upstream forced our hand

well, "our"

so I can't use a self-signed cert?

I followed the guide in the readme

clarkfischer: what do you know of SSL/PKI?

(yes, you can use a self-signed cert, it just needs to be built correctly)

I know there's a certificate chain, and that if it's self-signed, that chain doesn't go anywhere useful

do you know the term "CN" (or Common Name)? (in the PKI context)

Can't say I do

kk, one sec

the .crt doesn't have a CN section

Beginner Q: is there a monolithic jar anymore? I found 1.1.9 but it doesn't run well.

kewball: I don't believe so... Everything is now executed via "bin/logstash," and there's no more agent, etc

clarkfischer: so here's the deal

OK. THanks.

clarkfischer: you try to connect to foo.com via TLS/SSL, on some port

it presents you with a certificate, which you attempt to "validate"