#logstash

but my config is built modularly with puppet and i'd rather not surround all of my other outputs with conditionals :-/

it is: date {match => [ "event_timestamp", "ISO8601" ]} on logstash shipper

guess i'll have to though, huh?

looking to test out filebeat but am confused about this step: https://www.elastic.co/guide/en/beats/filebeat/...

<http://tinyurl.com/o9daqh6>; (at www.elastic.co)

do I run that on the ES cluster?

Seventoes: I think it's the only way - oterwise it will output to every output you have

BaM`: okay it seems like it's not having any trouble connecting to elastic search, but nothing's coming /in/.

thither: check your shipper logs

perhaps add an output { stdout {} }

just to be sure

BaM`: I did add that as an output already for debugging. Netcat seems to be able to connect :/

I guess OSSEC is the issue

I typed "foo" over UDP and logstash lit up

that sounds promising

I guess I'll bother the people in #ossec now

Thanks for your help

np!

rastro: anything wrong with that date filter?

robak: can you paste the content of your "event_timestamp" field ?

robak: assuming event_timestamp is set, and is set to an ISO8601 value, and the date{} stanza isn't conditional, and, and, and....

Title: ### Date example from Kibana's entry json: "event_timestamp": "2015-10-28T23: - Pastebin.com (at pastebin.com)

robak: use ISO8601_TIMEZONE instead

BaM`: in the match definition?

actually, no I don't think that's right

okay...

I thought that was what it was but grokdebug doesn't agree with me

BaM`: date{} isn't a grok format

so, any idea what's wrong with this config?

rastro: doesn't it still use grok date patterns?

BaM`: nope!

well, crap

BaM`: https://github.com/logstash-plugins/logstash-fi...

<http://tinyurl.com/pga2gy3>; (at github.com)

so what's the option? Use a Joda description?

BaM`: i wouldn't know if that input string is a valid ISO8601 timestamp, as determined by joda. maybe there's a joda debugger somewhere? :)

BaM`: i would also think that LS would log about the date format mismatch.

BaM`: so, in a quick test harness, that event_timestamp does work with the date{} filter.

BaM`: so, i would wonder if the conditional is being met.

so, this can't work the way I hoped?

robak: match => [ "event_timestamp", "yyyy-MM-dd'T'HH:mm:ssZZ" ]

lets try that

wont 'T' be treated literally as 'T' instead of T?

WFM: http://pastebin.com/yTXCgX9d

Title: Logstash startup completed 2011-11-11T23:59:59+00:00 {"message":"2011-11-1 - Pastebin.com (at pastebin.com)

why the Z at the end?

no you need to quote the T because it's not a joda value

read up on http://joda-time.sourceforge.net/apidocs/org/jo...

<http://tinyurl.com/mcdr8>; (at joda-time.sourceforge.net)

check my pastebin - it works fine

but why the Z?

timezone

and if there's no need to quote, then it should be match => [ "event_timestamp", "yyyy-MM-ddTHH:mm:ssZZ" ]

right?

no, you need to quote

otherwise the date filter will barf because it doens't know what a T is in joda syntax

anyone here run LS 2.0 yet?

BaM` has been trying to

BaM`: the example with the T makes it through the ISO8601 format just fine...

I'm having a heck of a time figuring out my new config file foramt

All I get is this: Error: Something is wrong with your configuration.

rastro: you talking grok or date?

torrancew: date{}

I should really finish the joda debugger I was working on

debugging date{} can be a real PITA

i took robak's sample event_timestamp and ran it through just his date{} and it works.

no kidding

rastro: so, am I doing something wrong?

rastro: oh - heh

I didn't test his original

torrancew: that would be awesome - I hate debugging date {} too

hey guys, that csv filter may be relevant

he parses a field called "event_timestamp" *after* the date filter

damn - I didn't notice the scrollbar ...

swap date <-> csv filters then?

wait - the whole config is important??

possibly. I can't tell what else may be relevant, not enough prior context

BaM`: I tried your example, and it doesnt work

BaM`: data from yesterday is with today's index name

ok, found the lats of my problems, LS2.0 running finally

robak: what does a raw input line look like before logstash gets it?

torrancew: its a csv

yeah you need to run the csv filter before the date filter

otherwise, how will it parse the event_timestamp field before it exists?

BaM`: how do I order that?

swap the lines around

:)

lets try that

BaM`: it works!

you're awesome guys, thanks a lot for your help!

+1 internet points for torrancew

crap, now document_id stopped working...

I've that in logstash-indexer output for elasticsearch: document_id => "%{conviva_session_id}"

it used to work before, and it doesnt now...

robak: paste us an updated copy of your config?

hello

I'm using rss-input plugin and some rss feeds getting imported fine but some just gives me an error

Error: undefined method `force_encoding' for nil:NilClass {:level=>:error} A plugin had an unrecoverable error. Will restart this plugin. Plugin: <LogStash::Inputs::Rss interval=>1800, url=>"http://www.netbsd.org/support/secu...;, tags=>["rss"], type=>"netbsd", debug=>true, codec=><LogStash::Codecs::Plain charset=>"UTF-8">> Error: undefined method `force_encoding' for nil:NilClass {:level=>:error}

<http://tinyurl.com/pb747zy>; (at www.netbsd.org)