#logstash

when doing an if in a filter and the field may contain an array how do you make sure you just switch on the first element?

Im confused: why would you use a "date" filter versus specifying "timestamp" in your grok pattern?

when groking an input that includes the timestamp string, do you normally exclude it afterwards to clean up the log, or keep it in the message?

__topwobble: great question!

__topwobble: so, grok's job is specifically to pull out text and give it names, like %{NUMBER:whatever} matches a text for numbers and calls the match "whatever"

at this point, it's still just text and has no intrinsic meaning to a computer (say, as a time concept)

Like, under the hood, if you ask a computer what time it is, it'll probably give you some text back, but there's an underlying data structure that allows the computer to do operations on that time value

like adding minutes to a time, or formatting it in a certain way, etc

the date filter takes a field with a time text and, based on your configuration, will parse it into something Logstash knows is "time"

so after you use the date filter, logstash will know what the time meaning of the text "Feb 27 19:43:00" means

ok. but why not just include "timestamp" in your grok filter pattern?

you can, but it's still text at that point

by text, I mean some characters with no real meaning to a computer

even if I call it "timestamp"?

like "Feb 27" to a human who speaks english knows this means February 27th

but to a computer it's a string of text with 5 characters, F, e, b, space, 2, 7

even if you call it timestamp

it's still just a piece of text named "timestamp"

ah. i thought "timestamp" was special

so you use the date filter to tell Logstash to take this text and parse it

here's what I am dealing with: https://gist.github.com/objectiveSee/49454a38b3...

yeah, I admit we don't document it clearly

<http://tinyurl.com/kqg3soy>; (at gist.github.com)

im matching timestamp in every pattern right now

ahh ok cool!

So if grok is successful, you'll have a 'timestamp' field with something like "2015-03-01T19:43:00Z" or something as the text

you'll want to use the date filter to help logstash know what (conceptually) the time of teh event was

otherwise the default time (@timestamp field) is the time Logstash received it

which isn't correct in most cases, so you'll want to use grok (like oyu are) and date filter to tell Logstash, "No, <this> is the time the event occurred"

in my example, "timestamp" is not getting set correctly, but seems to match

your date filter commented out has the wrong pattern

TIMESTAMP_ISO8601:timestamp will match a the TIMESTAMP_ISO8601 pattern

and your date filter: dd/MMM/yyyy:HH:mm:ss Z is the wrong pattern

you probably want match => [ "timestamp", "ISO8601" ]

(we provide a shorthand for the actual pattern that matches ISO8601)

my date filter is commented out

yep

so... im confused :D updated the GIST with sample input and its output. Its got the wrong timestamp value

hypothetically, I'm saying, if you uncomment your date filter, it won't work because it's using the wrong pattern

ah, right

"timestamp" => "2015-03-01T21:24:21.043Z", <-- looks correct to me

but to tell logstash "This is the real time of th eevent" you have to use the date filter

to set this field: "@timestamp" => "2015-03-02T03:48:29.664Z",

I Know it looks like a piece of text, but that's logstash just printing it in a way that you can read

there it is! works now

I thought I need %{ after the date match

"ISO8601" versus %{"ISO8601"}

hmm strange. I need to specify "timestamp" in both the pattern AND the date filter. I would have thought just the second

just read the name of every repo in github.com/logstash-plugins. Awesome stuff to come in next version it seems

__topwobble: yep!