#logstash

es is barfing when I tell it that the type is date unix_millisecond

er, epoch millisecond

have you tried epoch_seconds?

because that sure doesn't look like milliseconds from epoch. it's seconds with a fraction

es doesn't seem to like that either

I wonder if it is too many decimal points?

possibly. any error message?

just invalid format

"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [ts]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"1490800999.901811\""

(template is "ts" : { "type": "date", "format": "epoch_second" }, )

hm. often it prints where it failed parsing

which may shed light on this, but meh.

ES says maximum precsion is SSS

so I probably need to drop a few decimal points. Mutate?

hmm don't see a usable mutate option

well, if you're using LS in the first place I'd probably make that a proper date with a date filter

hah, yeah I stopped fighting it and let LS work natively with match => [ "ts", "UNIX" ]

eyeballing the comparison it looks to be working and setting the target @timestamp properly. I'll wait and see as load increases if there's a time discrepancy

but it should be now using the ts field as the @timestamp, which is the goal to retain sync

although it would be nice to have ES display the 'ts' original field properly.. that's what I was trying to get accomplished

Anyone have any help on my issues regarding unit tests?

Hello! I have a simple question about using logstash, but I can’t seem to find the correct online resource to educate myself

Essentially, I have a single field that contains JSON, and I’m hoping to decode that field, and create fields for each of the individual values in that JSON document

any help would be greatly appreciated. thanks!

Can Logstash update or remove/add a document if one with the same id already exists? Basically I want to track some features of browsers and only want 1 doc per unique user session to save space.

bnason: i believe you'd have to use custom document_id and update action mode with upsert enabled

thanks, ill look that up

Hello, I have a probably very basic question about the logstash-input-github plugin. This plugin requires an IP and a Port to connect to Github. How would I find this information? I have the URL I hit to use github, I have the API as well but none of this is done with an IP and port?

jack7238: did you read the explanation of that plugin? it accepts github webhooks; that is to say, LS launches a listener on the IP + port, and you tell github what URL to hit LS at

Right I think I just put that together. Sorry, still pretty new to logstash and first time with this plugin. Great tool.

no worries

has anyone run into / resolved the issue in logstash where it takes very long to execute a conf file from command line? (/usr/share/logstash/bin/logstash -f file.conf) ... takes like 5 minutes before it actually runs

msbh: Install some entropy gatherer, like haveged.

bjorn_: worked like a charm thanks

:)

'workers' isn't very well documented in the ES output plugin for logstash