#logstash

hmm. is there a way to seperate out a field as its own event?

I'm storing tweets, and I was thinking, since retweets contain the original tweet with updated like/retweet counts, it would be nice to update the original tweet document

I am not sure if there is a plugin that does that

but it would be easy to write your own to yield new events

There is an issue with the ruby filter which does not allow you to yield events (otherwise you could do that with that)

However, updating does not require you to create a new event. e.g. Elasticsearch ouput can update previous events based on the document_id I believe

pandaadb: yeah, but the original tweet is in the retweeted_status field of the retweet. I would need to split this out somehow

can you show me an example paste?

(of the retweet)

you can likely determine if a tweet is a retweet and then strip the retweet down to only contain what you want, and then update the document with that info

<http://tinyurl.com/jyvp3hj>; (at dev.twitter.com)

hah okay I see. Delete fields will invole a lot of fields

let me try something real quick

pandaadb: but what I want is to store both the full retweet and update the original tweet

hm okay. In that case I believe you will have to create a filter and yield the according event yourself

so copy the value of the retweeted_status key, and use that as a new event

time to learn ruby then I guess :)

<http://tinyurl.com/z2qf5wh>; (at stackoverflow.com)

it is fairly easy to do so. You create a new event, copy whatever you want into it and yield it

then your output can simply handle your custom event differently for the upsert

i recommend clongin an existing filter

for example: https://github.com/logstash-plugins/logstash-fi...

<http://tinyurl.com/zgqtc2c>; (at github.com)

because the elapsed filter also yields custom events, so the necessary code is already in there for you for the taking

pandaadb: cool, thanks for the tips

LotR it appears that you can do scripted updates with ES output

this might be what you want. So you could store the original event, and then have a scripted update depending on what you need

what is a scripted update?

<http://tinyurl.com/zdhqbxu>; (at github.com)

it executes a script I think instead of inserting the entire event

so essentially you can control what you want to update in your original event i think

Only just saw it so not 100% sure :D

<http://tinyurl.com/qahgoos>; (at www.elastic.co)

So the way it looks like LotR, the update API of ES can execute a script instead of just doing a usual upsert

So in the script you have access to the original event. You defined the document Id to find it, and create a groovy script that updates the values the way you want it to

and then simply run it. Might be worth a try :)

And by the looks of it it can do anything (create new fields, modify existing fields, overrite stuff etc)

threre is some useful info in here: https://github.com/logstash-plugins/logstash-ou...

<http://tinyurl.com/h983vrk>; (at github.com)

Hi, still looking for conceptual advice. Where would be better to introduce grok filtering - on collector nodes, or on indexing nodes. Using Redis on indexing nodes for buffering.