#logstash

the video is like 3 years old.

so to get to the point, could you make one example how to use ‘custom pattern’ and match something, I think I will understand it

(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601})

20160829173716

I have a rsyslog customization with timestamp

$template ConvertToTimestamps,"%timereported:::date-mysql% %HOSTNAME% %syslogtag% %msg%\n"

is there a “date-mysql” pattern already ?

as I figured only this solution: %{YEAR}%{MONTHNUM}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}

as for ‘custom patterns’

I can’t figure a way how they work i grokdebug.herokuapp.com

%{YEAR}%{MONTHNUM}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND} %{SYSLOGHOST} %{SYSLOGPROG}

now next for queueid I would like to use custom pattern and include that

8F1C9404FEDE

# common postfix patterns

POSTFIX_QUEUEID ([0-9A-F]{6,}|[0-9a-zA-Z]{15,})

I am stuck after this, not sure why it doesn’t match

Works for me

Hello, there is translate plugin which can use nested yaml dictionary in filter, but is there a way to value of specific field inside it ?

this is my nested dictionary, in translate plugin: E001: {T001: {P001: asd, P002: qwe}}, can i have only value of P001 from logstash ?

right now i am only getting all content when i am sending "E001" message

bjorn_: how do I match next value ? queueid

Just like you have written

I entered your queue ID and your pattern in grokdebug, and it gave the expected result.

emm but when there’s a longer line

Try it

20160829173716 mailer-post10 mail4.tld.com/smtp[15831]: 8F1C9404FEDE: to=<suman@gmail.com>;, relay=gmail-smtp-in.l.google.com[64.233.167.26]:25, delay=879, delays=122/755/0.26/1.7, dsn=2.0.0, status=sent (250 2.0.0 OK 1472481435 o1si32335596wjh.238 - gsmtp)

Add the whole line in grokdebug.

That's the point with debugging.

%{YEAR}%{MONTHNUM}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND} %{SYSLOGHOST} %{SYSLOGPROG}

Don't paste it here

yeah and now I am banging my head against a wall

Use grokdebug

how do I match next ?

"next"?

the queueid

8F1C9404FEDE

With POSTFIX_QUEUEID ([0-9A-F]{6,}|[0-9a-zA-Z]{15,}) just like you wrote.

I am not sure if %{SYSLOGPROG} mess it up

but it doesn’t match

the queueid

What does SYSLOGPROG match?

alone ? or with every matcher?

With your current grok filter in grokdebug

Title: pastebin - mungustas - post number 3707894 (at pastebin.ca)

is there a way to access specific field from dictionary in yaml: E001: {T001: {P001: asd, P002: qwe}} from logstash ?

mungustas: As you can see, the parsed value of SYSLOGPROG is 'mail4.mlsend2.com/smtp[15831]'.

Agree?

yes

Then in your filter, add the POSTFIX_QUEUEID pattern after that. But you will need to look closely at your log line, because there are special characters you need to consider.

maybe there’s a Tab char ?!

You haven't even added POSTFIX_QUEUEID so perhaps you should start doing that

well I did but then it doesn’t match anything

I pasted in the pastebin until it matches

It still matches SYSLOGPROG etc, right?

no

It does when I try.

No Matches

Using your pattern and log file

Ah, right.

%{YEAR}%{MONTHNUM}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND} %{SYSLOGHOST} %{SYSLOGPROG}:\s+%{POSTFIX_QUEUEID}

that’s where I am banging my head :)

\s means spaces (regular space, tabs, everything)

+ means one or more

alright!!

And don't forget the colon after %{SYSLOGPROG}

yeah those there my barier

first I didn’t match the colon and then didn’t know I needed to look for special chars (tab)

thanks bro!

It's probably not tab, just multiple spaces.

maybe

When you're debugging with grokdebug, add one new pattern at a time and make sure it works. Match the result to your pattern/filter.

In your case, the log line says mail4.tld.com/smtp[15831]: before the spaces, and the pattern match says mail4.tld.com/smtp[15831] (without the colon)

Then it's your job to add the colon

yeah I was wrong

do you know if there’s a syslogtimestamp already defined for date-mysql format ?

I see in example there’s (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601})

AFAIK, https://github.com/elastic/logstash/blob/v1.4.2... is the pattern file included in logstash.

<http://tinyurl.com/ow22gsm>; (at github.com)

Everything else must be added by you.

maybe I would not need to write all those year,month,day etc ?

ah ok I see there isn't

I guess I just should place it in custom patterns and use it as variable in match

I am just not sure I have to write all those patterns myself

<http://tinyurl.com/jth925q>; (at gist.githubusercontent.com)

# Postfix wrap em up

PF %{POSTFIX} (?:%{POSTFIXSMTP}|%{POSTFIXANVIL}|%{POSTFIXQMGR}|%{POSTFIXBOUNCE}|%{POSTFIXCLEANUP}|%{POSTFIXSMTPD}|%{POSTFIXREWRITE})

it’s like matching it all in one place or something ?

The | matches any of the given pattern into the "PF" field

so I guess I could do something like match the beggining like timestamp etc and then use “PF” to match everything else postfix related

That's probably what that pattern file allows. I didn't read it.

Hi, when executing the config-check: /opt/logstash/bin/logstash --configtest , i'm running into an error

when executing the config check file by file, it's ok..

It probably helps if you show the error message.

The error reported is: undefined method `+' for nil:NilClass

Mhm

strange, right?

michel_: i'd combine all individual configs to a single file and run configtest on that

Logstash will merge all the conf.d files, probably resulting in something that's not seen when you check one by one.

Exactly

cat * > /tmp/newconfig

ok, thanks. I'm going to look into this a bit more and post you my results!!

# ---------------------------------------------------------------------------------------------------------------------------------------------- # FILE INFORMATION # ---------------------------------------------------------------------------------------------------------------------------------------------- # Filename : 102-input.GOL.conf # Package : Base configuration # Owner : ING OIB Tooling # # -------------------

no goor result..

no good result

please don't post stuff like that

use gist/pastebin/etc

i'm not sure if this works for me

is there documentation for this usage?

What usage?