#logstash

jsandoval: plus you have no output section :)

when does logstash set its own @timestamp on incoming json logs? I'd like to use that field to store the server's timestamp and have it preserved

DanielHolth: if by "server's timestamp" you mean that from original logfile, then use the date{} filter to overwrite @timestamp.

rastro problem is @timestamp is the name in the json logs apache has been generating, so would the date filter happen early enough in the chain?

if @timestamp is the generated log, it should "just work" if you're using the right codecs

s/is/is in/

DanielHolth: ok, so you have the log file timestamp already. What else did you need?

rastro I'm sending logs to redis as json strings, then decoding them on the log server with redis-input codec=json, and @timestamp gets overwritten

DanielHolth: from what i've heard, LS will leave a @timestamp value alone if it comes from the input...

DanielHolth: what are you seeing, that suggests it's being overwritten?

I finally got it working!

I pointed the file input at a log file with several days of back logs, but nothing before "right now" appears in Kibana

<http://tinyurl.com/m57ec9h>; (at gist.github.com)

in case anyone else needs it

... trying a separate json filter rather than the builtin redis codec

DanielHolth: what does your @timestamp field look like?

looks like the separate json filter works

maybe the redis codec=json mechanism does not work

kireevco: I think someone added support for taking in headers from rabbitmq messages recently, but I don't remember

afk for a bit

Hi guys, I am trying to match gid=8 with a grokpattern like this GID (gid=|gid=%{INT:gid}) in the match I do %{GID:gid} and I end up with "god"; [ 8 ] and with gid=8

is god = gid

pjanzen: don't name your INT sub-pattern

good evening -- I'm having a lot of trouble to get the sign the CLA for PR 2210 (or, getting the check to agree with me that I did). Could someone support me with that?

ok... why not?

pjanzen: because then you wind up with two fields grok wanted to name gid...

%{GID:gid} would expand to %{(gid=|gid=%{INT:gid}):gid}, more or less

Ok, if I remove the sub-pattern I get INT [ 8 ] and I do not know what it is..

pjanzen: where are you doing this work at?

and what did you change it to?

I am testing at http://grokdebug.herokuapp.com/

Title: Grok Debugger (at grokdebug.herokuapp.com)

ok. now what is the pattern you just tried?

Let me create a gist....

thank you

lunch, bbiaf

<http://tinyurl.com/kodmr75>; (at gist.github.com)

pjanzen: I recomment you enable "named captures only" and 'singles" fwiw

and you are hoping that in the end, you'll have a field called "gid" that contains "gid=8", correct?

I want either null or just 8

well, that's not at all what you have currently configured

the pattern can be gid=8 or gid=

take a grok pattern %{FOO:bar}

that says "take the named pattern FOO, and store whatever matches it in a field called "bar"

in your first case, you were creating the "gid" field twice

Sure I understand that

once with just a number

and once with the whole text

what you REALLY want, is probably something like:

GID gid=%{POSINT:gid}

and then you'd just use it like %{GID}

that said, food time

oh, add a ? after %{POSINT:gid}

Ok but gid can also be just gid=

GID gid=%{POSINT:gid}?

yeah, the ? makes the posint optional

ok.. I'll try that...

torrancew, works like a charm... thanks..

oh, it was probably just sincedb start_position => "end" when I wanted "beginning"

One for weird questions: does the file output plugin hold onto the file handle between flushes, or does it refresh the file handle?

WAit... if I'm reading the plugin correctly (whack?) it opens a new file descriptor for EVERY event?