#logstash

I need to edit the logstash config file and change the value of a field dynamically while the pipeline is running, can I run a separate python script that regularly opens, edits and closes the file and have the pipeline continue with the changed fields?

Or will this be an issue?

Any help will be appreciated :)

yellow..

truthseeker1990: as I tried to explain last night, that won't work

I dont think we ever got to this point torrancew :/ Or I misunderstood what you were saying at the time

Can you just succinctly say why it wont work

Because Logstash won't pick up the changes to that file

Logstash reads the config files at boot

you'd have to change the file, bounce logstash

There is an automatic config reload option in the latest version of logstash

ah, I see

oh i remember, thats wjhy I came back to this option

(I did mention that I've not used 5.x)

I read about it last night

yeah

Apparently once this option is enabled, it scans for changes to the config file every 3 seconds

that may work, but it is very very hacky, and may be a bit brittle (particularly the programmatic editing of the config file)

Yeah it really is..I am wondering if maybe logstash might not be the perfect fit solution to my problem

I could write a python script that does what I need, and write the tweets to a S3 bucket. (which is what I wanna do ). Its just not as cool and integrated as using logstash -> S3 -> ElasticSearch

truthseeker1990: depending on the change needed, can you use the 'translate' filter? it can read from a file (which gets reloaded).

Hmm, So what I wanted to do was change the 'keyword' field in twitter plugin input in the config file so that I only get tweets that contain the trending hashtags. These change every hour. Its easy to do if you are working directly with the Twitter API but for some reason the only place in logstash where you mention the 'keyword' is in the config f

ile, and there doesnt seem to be any dynamic programmable way to set that field . The translate filter could work, except that it would still be reading a general sample stream of ALL tweets and then filtering them based on hashtags that I tell it to. It would kind of work, but it wouldnt get a lot of positive hits coz the sample stream is only a s

mall set of all tweets and what are the chances it will contain the hashtags that I want.

I am surprised that logstash gives you the abilty to read from twitter stream but doesnt give you a dynamic programmable way to interact with the twitter API through itself.

i've found that the world is full of disappointments :)

Wise words :)

truthseeker1990: My advice remains, write a simple thing to scrape twitter for the keywords you care about; your thing can get keywords from whereever you want it to (database, redis, disk, go wild with it)

then your thing can feed data to LS for further processing

via a file, or a socket, or whatever generic mechanism you like

Hey torrancew, would I even need LS if I can scrape the tweets? Coz I can do the scraping pretty easily.

IDK what you're doing with the data

Ah I am just feeding it to a S3 bucket

if you're trying to further process the data, LS may be a really nice thing to use there

or if you want to rely on LS's state-awareness or something, etc

(you'll still have to solve the twitter side of that, but LS could handle the s3 writing, if that's useful to you)

Yes! That would be useful.

One last question,assuming I wrote a python script to scrape the tweets, do you have any advice on how to feed the stream to LS? I dont think theres a programmable way to interact with LS

you'd use some generic mechanism to pass the data

a few options that come to mind:

write tweets to a file, logstash uses a file input to read them

truthseeker1990: i love using files for external scripts that gather data. it's the easiest least-common-denominator.

write tweets to a socket/http endpoint, logstash listens on that thing to receive them

write tweets to redis or something, if for some reason that is easier

rastro: is there an "exec" input type? if so, is it useful here?

(that you know)

torrancew: there is an exec. always felt like a hack to me.

I could see that

Ok cool, Thanks. I think I might just write to a file, seems like the path of least resistance.

to me, files beats sockets because it handles when LS isn't running.

agreed

truthseeker1990: if you write them as json, it'll flow in very easily. i do this with snmptraps.

and for situations where you have a lot of indexers you want to fan the data out to, I think a redis-like thing still beats sockets

torrancew: i had one of those, too. i would ftp data down, have that local logstash round-robin it out to three redis instances, and then had three distributed logstashes reading it from redis.

hi guys, good day to all

‘(\n%{GREEDYDATA:parts_message})+’ if I have this, I have one or more… but the frield name is been overwritten and I am just getting the ‘last’ message. Any way to either: each iteration has its down frield name OR turn it into an array?

frield name = field data

http://stackoverflow.com/questions/24605331/log... just like this guy

<http://tinyurl.com/llj6ahy>; (at stackoverflow.com)

Hi all , Could you please let me know how to check if the logstash is sending data to the elasticsearch?

rark: Look for indexes, either with Kibana (or something similar), or with e.g. curl.