#logstash

coreyfinley: can you hit the host with a browser on port 9200 and use that username/pw?

yes

Before going down the route of elasticsearch_http, i tried just using the elasticsearch output but got transport errors

input redis to output stdout works find so the input isn't problematic

maybe the codec is part of the issue. I've never put auth on my es cluster. i use the elasticsearch output with protocol => http, but i don't think that does auth at all.

I tried without the codec as well :/

and that's correct, the protocol => http doesn't support auth

looks like there's an ipwhitelist on the es side, and a way tot turn on http.basic.log, maybe double check those and see what error is tossed?

whoops, the ipwhitelist is to turn off auth for a single ip...nm that, ignore me.

#logstash

I have a small question. I am trying to make a tiny output to mysql

So I chose mysql2 gem

whenever I try to require in bin/logsatsh rib i get the following error

LoadError: no such file to load -- mysql2 from org/jruby/RubyKernel.java:1085:in `require' from file:/usr/local/Cellar/logstash/1.4.2/libexec/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55:in `require' from (irb):1:in `evaluate' from org/jruby/RubyKernel.java:1121:in `eval' from org/jruby/RubyKernel.java:1521:in `loop' from org/jruby/RubyKernel.java:1284:

sorry LoadError: no such file to load -- mysql2 from org/jruby/RubyKernel.java:1085:in `require' from file:/usr/local/Cellar/logstash/1.4.2/libexec/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55:in `require' from (irb):1:in `evaluate' from org/jruby/RubyKernel.java:1121:in `eval' from org/jruby/RubyKernel.java:1521:in `loop' from org/jruby/RubyKernel.java

LoadError: no such file to load -- mysql2

from org/jruby/RubyKernel.java:1085:in `require'

(sorry my first IRC experience ever)

please don't paste text like that in here, use a service like gist/pastebin/etc as it makes it easier to read and help

sorry for that, here is the link

<http://tinyurl.com/olk934l>; (at gist.github.com)

is there a limitation on the gems we can use for making outputs?

yahiaelgamal: wtf are you trying to do? use mysql as an output? you can't. i mean, maybe you could do some command line foo with exec, but there's no mysql output (that i know of): http://blog.fernandobattistella.com.br/2014/10/...

<http://tinyurl.com/ksr5oux>; (at blog.fernandobattistella.com.br)

and yes, you can't just grab a gem and expect it to work, you need to write an output plugin.

That's exactly what I meant by making a mysql output. I didn't use the gem in the config file ofc. I used here https://gist.github.com/yahiaelgamal/38ef37f35f...

<http://tinyurl.com/pku5ckm>; (at gist.github.com)

I read the blog post, but if the problem is in the jdbc, why not use another connector

yahiaelgamal: are you installing everything with ports on a mac?

pack: I installed logstash via homebrew on mac

pack: and using rbenv. nothing special

yeah, i wonder what that does for gem pathing, and if it uses the system paths or if you have to toss it in the vendor bundles (/opt/logstash/vendor/bundle/jruby/1.9/gems/ on cent6 boxes using the rpm)

(sorry, i don't think i'm being helpful)

pack: I checked in jruby/gems directly on my machine, the mysql2 gem isn't there. I will try to add it there (not sure how to maintain this though)

I tried and it didn't work either. same error

pack: Thanks man appreciate it

is there any way to run multiple input threads?

for what input

you can have multiple LS worker threads, and some outputs support the same, not sure on inputs though

each input get's it's own thread, http://logstash.net/docs/1.4.2/life-of-an-event

Title: logstash - open source log management (at logstash.net)

pack: yeah, I saw that. I'm considering duplicating the input

I believe messages are getting pop()'d off by some mechanism, so I shouldn't get duplicates

redis has it's own thread thing. what input are you using?

rabbitmq

<http://tinyurl.com/ptbfz9r>; (at logstash.net)

"This is the same as declaring the input multiple times" awesome <3

I wonder what it does with file globs

like for the input type file...one per file...one per statement...

Is there a reason that grok patterns don't usually convert the fields they extract to their natural type? E.g. the pattern for HAProxy log's 'backend_queue' field is %{INT:backend_queue} instead of %{INT:backend_queue:int}

Which means I can't simply do term_stats on those fields without converting them myself first

martbhell: thanks!

Performance would be my obvious guess, resp. a "only convert what you need" philosophy. But would that really even matter in terms of performance compared to everything else logstash does during the filter phase?

Also: How should I go about converting these fields? Add a bunch of mutate { convert => {}} filters? Fiddle with the ES mapping? Or are there other options?

how would one parse a timestamp that looks like [02/Nov/2014:06:26:55 -0500]

I am having trouble with the [ and ] characters apparently

deruke, that format corresponds to the HTTPDATE pattern: https://github.com/elasticsearch/logstash/blob/...

<http://tinyurl.com/ohowabk>; (at github.com)

ok cool - it was failing on the debugger which means the problem is likely behind the keyboard

Can't you just write your grok pattern so that it extracts the date from within the [ ] to a 'timestamp' field the same way the COMMONAPACHELOG pattern does it?

So, the HTTPDATE pattern does not include the [ ] themselves, that's handled in the COMMONAPACHELOG

So for example, use \[%{HTTPDATE:timestamp}\] as pattern and [02/Nov/2014:06:26:55 -0500] for the input on http://grokdebug.herokuapp.com/