#logstash

dcaro: no output on elastic log/console even when output does reach it (no filter mode)

what does this mean? log4j, [2014-04-30T11:25:23.740] DEBUG: org.elasticsearch.discovery.zen: [logstash-logstash-4538-2010] filtered ping responses: (filter_client[true], filter_data[false]) {none}

so this works, which means logstash receives the data ... /opt/logstash/bin/logstash -vv -e 'input { udp { codec => msgpack {} port => 1717 }} output { stdout { codec => rubydebug }}'

can anyone think of reason that grok parsed messegas fail to reach elasticsearch? without grok filter (or msgs with _grokparsefailure tag), everything reaches elastic just fine

the conf file has output { elasticsearch { embedded => true } } ... but I don't get anything in kibana .. and /var/lib/logstash/data/ doesn't grow

Interesting that there is no post-processing options. I'd like to either move or delete a file after logstash has processed it.

qru: you can treat the tagged messages with another grok rule or drop them as you like afaik

argh, and "/etc/init.d/logstash-web start" says failed but it does run

qru: I'm very new but it seems like logstash keeps monitoring the file so for new entries, so it's never finished with it

yfried: Depends on the log file. In this case the app that creates the "log" file dumps contents into a new file each time.

Continuous log files are dicey.

You have 3 processes really: The writer, the reader, and the cleaner

Making sure all those are on the same page can be difficult.

Any one got experience with logstash and centos 6.5 ? like which java to install, other packages I might need?

cmdstation: I'm running it there

cmdstation: just java-1.7.0-openjdk

ok Ill try. I did yesterday and got some bug with the agent...

But gonne give it a try now

cmdstation: you're using repos, or tarball?

tar

I'm using repos, kinda feels easier

hm

ill give that a shot

Also if using repos remember you probably also need logstash-contrib package

why am I seeing data on elastic-head (localhost:9200/_plugin/head/) but not on kibana (localhost:9292/index.html#/dashboard/file/logstash.json)

yfried: clear browser cache?

I keep getting the same error: undefined method `+' for nil:NilClass

I know its known

but cant seem to get it fixed on centos

viq: no help

viq: it's wierd. I thought I wasn't getting gork parsed data. turns out I'm just not seeing it in kibana

yfried: wrong index or something? Wrong fields?

viq: what do you mean? I want to see all the data on my server. I'm seeing nothing right now in kibana

I see a refernce to config.js file. where is it?

yfried: should be in main dir of kibana, I think

viq: where is that

yfried: wherever you put it. Or are you running the integrated one?

morning

are there packages built for the logstash-forwarder aka lumberjack for ubuntu?

viq: I think it's the 2nd one, cause I didn't install kibana

I was curious if there was a default port most people use... it said there was no default in the docs but was curious if some unofficial standard

just ran logstash... web

yfried: I use git checkout and servie it via nginx

elasticsearch embedded doesnt seem to start correctly, what could be the problem? Here's my config: https://gist.github.com/Meai1/52aa618a92e321337697

Title: gist:52aa618a92e321337697 (at gist.github.com)

and I start the webui like this: logstash web --port 9292

viq: dcaro|lunch: seems like I was looking on too small a window. parsed files have an earlier timestamp and I was filtering them out

nvm now it works, after waiting 5min..

yfried: oh, so it's working now?

Would it be a good idea if i pull the 1.4.1 version from git to evade the bug atm: undefined method `+' for nil:NilClass

hi, I'm having some trouble connecting to logstash with the forwarder, does anyone know what might cause an i/o timeout? I wish I cold get a more descriptive error

dcaro: sent you link in PM

dcaro: could you think of a way to have the shipper shut down once it finished parsing all the files (files are no longer logging new data)