#logstash

I've been struggling for a while to parse datestamps like this: 1422585233

.00 into ISO8601. I'm storing it as "timestamp" of type NUMBER

ICantCook: that's a unix epoch value?

rastro: Yeah

This is what I've tried so far: http://pastebin.com/FeVfvrJq

Title: filter { if [type] == "squid-access" { grok { match => { "message - Pastebin.com (at pastebin.com)

date{} will match UNIX with an epoch value/

SQUIDACCESS %{NUMBER:timestamp}%{SPACE}%

I've tried that, as per my pastebin

ICantCook: but you don't have an ISO8601 format field, you have a UNIX format field.

but not seeing it in the logstash stdout

match => [ "timestamp", "UNIX" ]

oh, I thought that was where I put the desired output

I want to convert it that ISO format

ICantCook: no, that's the input format.

ICantCook: date{} converts it into a, um, date./

ICantCook: by default it'll stick it in @timestamp, but you can choose a different destination.

rastro: awesome, that worked :)

ICantCook: great.

I am getting filter worker exceptions in logstash 1.4.2, I am trying to use a grok to define a field, check the field value and set another field depending on the results.

Title: filter { if [type] == "mongos" { grok { match => { "message" => " - Pastebin.com (at pastebin.com)

if I comment out lines 25 thru 33 I don't get the exceptions

the exception messages are in the pastebin

I'm planning to run curator on every older than 60 days, but would have a backup of this first. Is there a way to dump all the data >60 days old into a flat json file(s)?

ICantCook: you'd need to pull it out of elasticsearch

andrew[andrboot]: is there a better solution than what I've suggested?

ICantCook: not sure tbh, if you could archive all x entries on 1 node or 2 nodes that would probably work?

goal is to retain data forever in tar.gz'd encrypted files in amazon s3

aaah

yeah, then one day if I'm asked "What happened 3 years ago?" I could restore the apropriate data and answer

I might give this a go: https://github.com/taskrabbit/elasticsearch-dump

Title: taskrabbit/elasticsearch-dump · GitHub (at github.com)

I like the logo

ICantCook: nice and shiney.

andrew[andrboot]: it's working perfectly as well

I assume there's a way to import raw json to logstash

(restore situation)

ICantCook: nice :)

ignarps: you get this error because of this line: } else if [pcursorShardTargets] > "1" {

ignarps: the error message is not helpful, sorry about that

ignarps: the error means that you have no field named 'pcurserShardTargets'

ignarps: try:

} else if [pcursorShardTargets] and [pcursorShardTargets] > "1" {

this will check to see if the field exists first

before trying to compare it with "1"

beloved nilClass