#logstash

does anyone here have experience of deduplication with the document_id attribute of the elasticsearch output? I'm using lumberjack to transport my logs and have document_id => '%{offset}' in my ES output, but I'm still seeing dupes :(

Anyone have any ideas?

offset doesn't sound like the best way to dedup

i tried deduping before, but hit too many bugs, and stopped :)

hmm, bugger

maybe stuff improved over time

but one of the main things i remember about deduping was performance impact

there is very little documnetation on the document_id attribute

"The document ID for the index. Useful for overwriting existing entries in Elasticsearch with the same ID."

is basically it :/

the suggested way to dedup is making a checksum of the data, and that costs lots of cpu

do you have a link to docs for checksumming events? I'd like to at least give it a go to assess the perf hit

I'm happy having more tin to handle the checksumming and getting cleaner data in ES tbh

lookg at fingerprint checksum/fingerprint filters

great thanks :)

so what to do... i'm not using ipv6

jopz: are you using elasticsearch embedded or a seperate JVM instance for Elasticsearch? Your configuration suggests it's the latter.

jopz: Could you try this configuration and see if things improve? http://pastebin.com/yr0L9FNv

Title: suggested elasticsearch output config - Pastebin.com (at pastebin.com)

Hellow, is there any way to see which logs are being sent by logstash-forwarders?

let me check this

losh: not worked

again getting same error

jopz: change 127.0.0.1 to ::1

k

oh, thought you were checking for me, jopz

:p

fuwan: no

Fuwan: I'd use tcpdump or tcpflow to see the logs that are being sent to the forwarder (assuming they're are being sent over a inet socket)

hmm

losh: no

same error

:)

jopz: Are you running Elasticsearch as a separate JVM instance?

jopz: Or was it running in embedded?

only one jvm instance

for all

jopz: Ahh, OK. In that case, the configuration I supplied disabled to ES. Set the option embedded => true in the configuration

jopz: In fact, the configuration is wrong for using the embedded version, I'll re-write it and send you the link

k

losh: send the config to my email id jaijopj@gmail.com

jopz: I don't hold much hope, but your can try this for embedded http://pastebin.com/4N7w6i9c

i'm leaving the forum no

Title: embedded elasticsearch example - Pastebin.com (at pastebin.com)

same error

bad luck losh

is it possible with logstash to convert unix time to readable format?

jopz: So it would seem. I'm out of ideas, sorry.

k

thanks

i will find some others

muranese: yes, use the date plugin.

losh: you mean filter? don't get how it works, in grok i add 'match => { "message" => "%{NUMBER:epoch}" }' can't find option for date filter

and to date fillter add this "match => [ "epoch", "UNIX" ]" right?

muranese: Yes, the date filter is a logstash plugin. grok is a separate plugin to date.

Title: logstash - open source log management (at logstash.net)

muranese: That looks correct to me, assuming your field name which contains the unix timestamp is called epoch.

hey, does logstash use the same version of ruby as my system configuration or is it it's own ruby version?

How is it possiable that a system I have shutdown is still reporing to logstash?It's 'exactly' every 60 seconds. http://i.imgur.com/mhYt27r.png

stasher: it's jruby, so it uses whatever version they bundled when they packaged the jar-file I'd think