#logstash

anyone know what this error means or how to fix it? http://dpaste.com/2QNTRC5

Title: dpaste: 2QNTRC5: logstash error, by soulbname (at dpaste.com)

<http://tinyurl.com/mygd3l6>; (at gist.github.com)

does anybody have another idea?

is there a way to ask logstash what it's plugin path is? this seems stupid since it recognizes the other plugins just fine.

It does that even if I don't set protocol.

rdobbs: it sounds like the cookbook messed up the install somehow, frankly

I'll be back online later this weekend

anybody else?

I don't feel like I'm doing anything special

rdobbs: It looks like it's specifying a library path?

-I/opt/logstash/lib

gregmefford: dunno, haven't started reading through the logstash code.

is there a variable to force logstash to look for plugins/libraries in the right location?

rdobbs: There is, but if other plugins are loading, it's probably not that. And also, I think it still loads from core in addition to the one you specify.

Can you gist your config?

"LoadError: no such file to load -- logstash/outputs/elasticsearch/protocol

any idea as to where it's looking for that? relative paths suck.

sure, there are a few involved because of upstart

Yeah, from looking at the code, I think it's a bug.

It looks like it's doing a raw require instead of using the Environment::plugin_path.

reallY? this must be the most common output

It looks like it would probably work if you happened to have the invocation perfect, but it's not as robust that way.

what is a "perfect invocation" ?

<http://tinyurl.com/ollfk85>; (at gist.github.com)

What happens if you run it without -I/opt/logstash/lib ?

not sure. it will take me a while to figure out how to do this 2 levels of complexity.

fuck upstart, by the way.

Wait, LS_HOME is in /var/lib/logstash? Or is it in /opt/logstash?

I recommend trying a simpler invocation to see if it works, then working back toward the more complex one with the memory tweaks and stuff.

look at my notce

notes. I've tried it both ways.

same error.

it takes 2-4 minutes to teast.

Try this: /usr/bin/java -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d

is there any way to make logstash startup in a reasonable amount of time?

this thing starts up slower than my

sigh. no target for my pun.

Don't use Java

:)

can I run this without java?

java makes pandas sad

Yes, you can just use the bin/logstash command from inside the source directory.

It runs fine under MRI

then why does everybody use java?

like by default?

hello logstash, does the conditional regexp parser support backreferences/

(caveat: it has to be a recent ruby, not 1.8.6)

hrm I'll try that next. it'd be nice to have a rapid test cycle

Java does a good job at larger scale, I think.

LoadError: no such file to load -- logstash/monkeypatches-for-debugging

when I run with the cli you gave me.

What if you cd into /opt/logstash and then just run bin/logstash agent -f /etc/logstash/conf.d

ooh progress, new errors.

Sweet!

kizzale: Sorry I missed your message. Can you clarify what you're trying to accomplish?

ok that fixes it

rdobbs: Horray! o/

So another thing that will help you iterate quickly is that if you only have a file input, LS will process them and then exit.

So you can use that to test out your Groks and stuff, just process a file from in to out and see if it works.

GregMefford: a horrificially complicated regex that is the best way i can figure to do what i'm trying to do; if ~= /gigantic horrific regex with backreferences/ { ....... but messages that are mathching that regex elsewhere are not getting tagged with the add_tag i put in that block, so it's not matching in logstash

thanks man, this makes me feel less stupid. been one of those long days.

one type of message going thru my logstash is totally blocking the grok parser, so i'd rather not grok those messages and just drop them, lest i block the filter (and processing dies)

time to go home and have the californian equivalent of an afterwork drink.

rdobbs: lots of drink?

rdobbs: Glad I could help, have fun!

unless.... is watchdog timer back in 1.4.x? i'm still on 1.3

kizzale: Ok, so it sounds like there are probably several ways to solve the problem you're having. Are you able to gist the relevant part of your config?

yeah, un moment

sorry, anonymizing some of the message

:)

I owe you a caliufornian beer equivalent

kizzale: NP, at your convenience.

Logstash forwarder is not sending logs to Logstash in my setup. These are the config files - https://gist.github.com/theharshest/284f9213c81...

<http://tinyurl.com/kgre36s>; (at gist.github.com)

My logstash nodes are behind load balancer, and logstash forwarder nodes are forwarding logs to load balancer

I'm having trouble parsing an RFC3339 timestamp. Is RFC3339 not the same as ISO8601?

GregMefford: any suggestions here?

harshjha__: What is the bahavior you're seeing? No log traffic is hitting the indexer(s) behind the load balancer?

GregMefford: you are right

GregMefford: and if I provide any input through stdin in logstash node itself, it can be correctly seen in output - stdout as well as elasticsearch

nemothe__: I'm not familiar with RFC3339, but I just Googled it and it seems to refer to ISO8601 as the only examples contained in it. Can you give an example of what the timestamps look like that you're trying to parse?

GregMefford: there are no issues in logs on other side. Also I can see messages like "Registrar received 9 events" on forwarder side.

*either side

harshjha__: What kind of load balancer is it? Are you sure it's configured to work the way you expect it to?

The timestamp is "2014-06-27T16:55:32.325Z-07:00" the issue it seems is RFC3339 has both the "Z" and the timezone offset. Grok isn't picking up the timezone offset so the entire thing failes

GregMefford: yes it is configured. It is elastic load balancer. I have my setup on AWS. I can see in the forwarder logs that it is connected to load balancer without issues. Load balancer is configured to take traffic at 443 and send it to 7286.

GregMefford: https://gist.github.com/anonymous/7060f0b6ff924...

<http://tinyurl.com/l2gs9sa>; (at gist.github.com)

GregMefford: Actually it might be a bug in golang

i kind of have a feeling there is somekind of congestion in logstash. my system has lotta resources. I still dont see logstash doing well

some one has some kind of perf charts?

GregMefford: bug in the library I'm using thanks

rc_: have you increased the # of filter workers?

nemothe__: Yeah, that makes sense that it would be confused by having a Z and an offset. You should be able to match it with the following Grok expression: %{TIMESTAMP_ISO8601}(?:Z%{ISO8601_TIMEZONE})?

thats' the big thing -- by default it only has 1 filter worker

if you have more cores, you can pump that up and significantly improve performance