#logstash

Hello How can i connect to ES cluster which has multi Nodes.

Hello How can i connect to ES cluster which has 5 Nodes with 3 eligible Masters.

connect with logstash?

Can anyone help me with his question? https://groups.google.com/d/msg/logstash-users/...

<http://tinyurl.com/of58g55>; (at groups.google.com)

did that bottom answer at SO not help?

What, the one from Ulrich?

that last comment at the bottom with the complete config

My SO question has been answered, I can parse the timestamp and location data correctly. Now I need to figure out how to step through the array so I can deal with multiple events.

The other question I have is about replacing "\\n" with "\n" here https://groups.google.com/d/msg/logstash-users/..., but one thing at a time

<http://tinyurl.com/pp5p7bc>; (at groups.google.com)

hi

parsley72: I may be missing context, but it seems wierd that you're having to do this \\n stuff in JSON. A newline in JSON is represented by a two-byte sequence \n

I had to replace "\n" with "\\n" to get Keen.io to accept it, might be their limitation rather than JSON's.

weird, though I don't know anything about keen.io

:(

But the array stepping is my primary problem right now

array stepping for what?

let's sayou have an array, what do you want to do with it?

<http://tinyurl.com/of58g55>; (at groups.google.com)

I want to step through the array of result[] and create a separate logstash entry for it

you want that to stay as 1 event?

or make multiple events?

Multiple

you may need a custom filter

Hi All, I'm planing to setup an logstash server for my AWS VPC, since It's a pain in the ass to maintain anything java based think I will use the docker image, and if it's already a docker container I'm planing to use the AWS ECS,so that part is cool i just have to figure out the config file ec2 connection but that's fine, I could live with that, but not sure about the database backend, elasticsearch is only available across EMR, but that also looks

paint-full but there redis does not seem to be the right one for this job, can you guys recommend something easy to use AWS PaaS database which is supported as an logstash output ?

The answer I posted on SO only handles the first event

parsley72: I blackhole SO on my browser, so I can't help there.

parsley72: but this ruby filter may help you get started

ruby { code => 'event["result"].each { |o| yield LogStash::Event.new(o) }; event.cancel' }

Sorry, warkolm was talking about the question I asked on SO

yeah no worries :)

parsley72: the ruby filter config above should take every item in the "result" and emit it as a new event down your filters

Does that filter go before the rest of my code?

so after that, you'd have N events

that filter goes before anything youw ant to process with those N events

thanks, i'll try it now

ruby { }, then date { match => [ "[keen][timestamp]", ISO8601 ] }

something like that

i've got that working already, thanks

Oh right, after the filter, I've got you

yeah

the ruby filter will split your one event into many

(or should, assuming I wrote it correctly)

Hmm, not quite: Exception in filterworker {"exception"=>#<NoMethodError: undefined method `each' for nil:NilClass>, "backtrace"=>["(ruby filter code):1:in `register'", "org/jruby/RubyProc.java:271:in `call'", "/media/tom/59873272-ced5-4687-84e0-fa42aff3eb9b/logstash-1.4.2/lib/logstash/filters/ruby.rb:38:in `filter'", "(eval):35:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/media/tom/59873272-ced5-4687-84e0-fa42aff3

according to your example json, you will have a field named "result"

the error you got indicates that you do not have a field named "result"

Yup, which is an array

ruby { code => 'p event.to_hash; event["result"].each { |o| yield LogStash::Event.new(o) }; event.cancel' }

try that, and paste the event that is output

the 'p event.to_hash' will print the event to your console

Trying it now...

anyone on the AWS integration question ?

That gives me: {"message"=>"{\"result\":[{\"keen\": {\"timestamp\": \"2014-12-02T12:23:51.000Z\", \"created_at\": \"2014-12-01T23:25:31.396Z\", \"id\": \"547cf8eb072719278f721f5d\", \"location\": {\"coordinates\": [-95.8, 36.1]}}}]}", "@version"=>"1", "@timestamp"=>"2015-04-29T05:30:35.753Z", "type"=>"json", "host"=>"APE01-AKLNZ", "path"=>"/home/tom/test6.json"} Exception in filterworker {"exception"=>#<NoMethodError: undefined m

parsley72: very strange, that looks ok

Ah, should the ruby filter come afte the json filter?

ooooh

let me rephase, it doesn't look ok

your event has a "message" field (normal) which includes unprocessed json

the literal json text

so basically, logstash just sees text, it doesn't know it's json

halt: question is very good, i'm also interested

right, so the first thing I do is run it thorugh a json filter

warkolm: r u there?

halt: EMR == elasticmapreduce? This has nothing to do with elasticsearch.

if i put the ruby filter after the json filter i get: {"@version"=>"1", "@timestamp"=>"2015-04-29T05:33:15.919Z", "type"=>"json", "host"=>"APE01-AKLNZ", "path"=>"/home/tom/test6.json", "result"=>[{"keen"=>{"timestamp"=>"2014-12-02T12:23:51.000Z", "created_at"=>"2014-12-01T23:25:31.396Z", "id"=>"547cf8eb072719278f721f5d", "location"=>{"coordinates"=>[-95.8, 36.1]}}}], "tags"=>["valid_json"]} Exception in filterworker {"exception"=

VPC is more about networking than it is about servers, isn't it?

parsley72: what's the exception?

whack: https://blogs.aws.amazon.com/bigdata/post/Tx1E8...

<http://tinyurl.com/peuju3t>; (at blogs.aws.amazon.com)

Exception in filterworker {"exception"=>#<LocalJumpError: yield called out of block>, "backtrace"=>["(ruby filter code):1:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "(ruby filter code):1:in `register'", "org/jruby/RubyProc.java:271:in `call'", "/media/tom/59873272-ced5-4687-84e0-fa42aff3eb9b/logstash-1.4.2/lib/logstash/filters/ruby.rb:38:in `filter'", "(eval):49:in `initialize'", "org/jruby/RubyProc.java:271:in `c

parsley72: bah, I wonder if there's a bug making it so you can't yield from ruby filter

No idea, Ruby's all Greek to me

.oO( cookbooks/herebedragons/recipes/greek.rb )

parsley72: no worries :P

parsley72: I filed a bug: https://github.com/logstash-plugins/logstash-fi...

<http://tinyurl.com/ldhnafb>; (at github.com)

in terms of getting this working for you immediately, I don't know if I have any quick fixes for you :(

(: thanks for trying

Ok I have to leave now, I leave the question open, and will check the botbot.me logs in case anyone have suggestions feel free to ping me thanks in advance

naptime