#logstash

my bad then

MrGuga: RELP is rsyslog's custom protcool, sorry for the confusion :(

google skills failing me =(

is there a plugin for 3195?

whack, ok, so is there a good example of something forked within a logstash plugin? I need to interval something, but the IRC plugin has the blocking pop.

says at the wikipedia that rfc 3195 is implemented in logstash, that's the source of my confusion

MrGuga: !

MrGuga: http://en.wikipedia.org/wiki/Reliable_Event_Log... is about RELP< not RFC3195

<http://tinyurl.com/mtj5chz>; (at en.wikipedia.org)

"While RELP is still not formally standardized, it has evolved into an industry standard for computer logging." is ... false

I don't know anything but rsyslog that uses RELP

logstash only supports it because rsyslog is shipped with RHEL

well maybe I could somehow use rsyslog to transcode from relp to redis, but i wonder if it's worth the time...

MrGuga, I've had no problem with strait syslog from my fortigates, ofcouse I don't expect 100%..

and, now that I think about it, it goes though some fortianalyzers (not my choice)

Is there anything I could do to make sure I'm not losing packets? it's UDP after all...

MrGuga, via Fortigates?

it might support syslog over tcp

how many fortigates do you have?

one cluster

whack: afaik, tcp only with RELP.

MrGuga: you mean rfc 3165, right?

MrGuga: for now, I have no great solutions. Logstash does its best not to drop udp :P

we can probably implement RFC3165 though

MrGuga, ok, looked at my fortigates -- If you only have one, then an Analyzer isn't going to do you any good. ;)

And, it's only UDP anyway

lol yeah

I don't see where you get the open for RFC3165 tho.

None of mine show that as even an option

whack: is it that easy?

coolacid: maybe it's only on cli

MrGuga: I haven't read the protocol (RFC3165) but adding support for new inputs in logstash is pretty easy (in my opinion, which is skewed because I write lots of them)

That's possible. I don't usually deal with the fortinet gear unless i'm on call ;)

whack: Well, for a first timer probably will take a couple of days to get it working. I can't afford it. I'll hope all packets are just arriving, if something gets suspicious I'll get another machine just to receive udp and queue to redis.

MrGuga: wasn't suggesting you had to do it, for sure I would expect a few hours or days to get something like this working on your own, especially the first time. :)

MrGuga: why would you queue to redis?

or the 6th time ;)

udp doesn't necessarily drop because there receiver drops it

btw, to grok a string followed by ":" should i use "%{NOTSPACE:field}:" ? or is there something better?

MrGuga: that's probably OK

alternate, if confusing, syntax for things specifically like that

(?<field>[^:]+):

captures [^:]+ to 'field'

where [^:]+ means 'any non-colon character'

whack: true... but since the receiver is a VM, i expect some I/O problems with high usage. Network is very well dimensioned so it wouldn't worry me.

whack: thanks

Ok - so new functions in IRC input "works" but because of blocking the stats don't fire without an event from the server

does anyone have any ideas on querying data like this: show me the top results of field_b grouped by field_a?

coolacid: you can spawn a separate thread if you wish

coolacid: what are you working on?

whack, yes, I need to figure that out.

<http://tinyurl.com/lyuhfdp>; (at github.com)

More things we'll use in DR, but figured it would be good for anyone that wants it ;)

if my message contains any macaddress, how can i grok it to a field? not all messages will have it, so should i just grok with the rest of the string as trash and remove failure tags afterwards, or is there a better way of doing it?

Hey, I've got what I think is a better syslog 5424 pattern match. Is anyone interesated?

rhollan: yes indeed

the community wants one for sure :P

that moment when you copy from git repo to working directory instead of the other way around.

Title: RFC 5424 syslog patterns - Pastebin.com (at pastebin.com)

I THINK I got the UTF8 stuff right.

But would appreciate others' testing it.

I

I've been using it on my logs, which, sadly don't have UTF8 messages or parameter values.

recall that some parts of RFC 5424 logs are ASCII and some parts are more general UTF8

coolacid: re: stats, I can help you work on that stuff if you want

actually, I think I got it.

Testing again - PR soonish

<http://tinyurl.com/kb2hk8c>; (at github.com)

Must dash off to get kids from school.