#logstash

hi

i have filebeat sending events to logstash (verified that events are catch by logstash)

however i am not able to end up with a "filebeat" index in elasticsearch

it did not result in localhost:9200/_cat/indices

output { elasticsearch { hosts => [ "localhost" ] } }

any ideas?

how is filebeat different from logstash and why its better?

f0s0: try specifying port. "localhost:9200"

or better logstash-forwarder

sol_, it is just a replacement for logstash-forwarder , a newer one

f0s0: yes i know logstash-fw is allready tagged as obsolete but why did it get repleaced?

sol_, I'm just starting with ELK, so I tried the last one

i didnt see that and installed fw, so now dont know if i should repleace it

however filebeat -> logstash is working

any clue if its compatible?

think it is meant to be, being made by ELK guys :)

sol_: what do you mean by "replaced"?

Norrland, same behaviour when replacing localhost with http://localhost:9200

why 'http://' ?

i am not able to debug whether logstash is creating index in elastic

oh

f0s0: try enabling debug logging in filebeat.

running filebeat with -e -v by shell

it is sending events

running logstash with rubydebug from shell

it is receiving events

problem is logstash -> elastic

hastebin your logstash configuration please. Someone might be able to spot something.

rubydebug gives information about input being working but not about output

ok, same when using localhost:9200

my conf is http://pastie.org/10591867

Xylakant: if you look on the github page of the forwarder it says: An experiment to cut logs in preparation for processing elsewhere. Replaced by Filebeat

my concern is , why rubydebug does not debug whether output to es is working, provided that is correctly debugging input from filebeat

sol_: yes. in the medium run, you probably should replace LS-FW by filebeat.

looks like

ok i understand that rubydebug is a kind of output, not a kind of log

how can I get a useful debug from logstash about its communication with ES? the generated .log just contains java stuff

f0s0: what does the java stuff say?

Norrland, IMHO nothing useful: tons of lines about buffers, mutexes, low level debug, 1 lines per sec

mkay

acht, found a line making sense, in relation with input event

{:timestamp=>"2015-11-30T12:02:05.303000+0100", :message=>"Flushing buffer at interval", :instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x712466e @operations_mutex=#<Mutex:0x9d5a8fc>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0xdea2d1e>, @submit_proc=#<Proc:0x5ddcbc82@/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.4-java/lib/logstash/outputs/elast

icsearch/common.rb:54>, @logger=#<Cabin::Channel:0x79ea6d8d @metrics=#<Cabin::Metrics:0x25342d73 @metrics_lock=#<Mutex:0x28c75d88>, @metrics={}, @channel=#<Cabin::Channel:0x79ea6d8d ...>>, @subscriber_lock=#<Mutex:0x1047dcbe>, @level=:info, @subscribers={12452=>#<Cabin::Outputs::IO:0x3e30a0b5 @io=#<File:/tmp/logstash-debug.log>, @lock=#<Mutex:0x2c8981e6>>}, @data={}>, @last_flush=2015-11-30 12:02:04 +0100,

@flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x2050ae5>, @buffer=[], @flush_thread=#<Thread:0x62672536 run>>", :interval=>1, :level=>:info}

sorry for the stuff

this lines mention ES output at least

but not so useful to me

which versions do you have? Logstash, Elasticsearch?

logstash 2.1.0

elasticsearch: Version: 2.1.0, Build: 72cd1f1/2015-11-18T22:40:03Z, JVM: 1.8.0_65

actually we had the stack already running in the past, but I am doing from scratch

is it expected that i create manually a index "filebeat" in ES via curl?

i already created the template for filebeat in ES, with curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json