#logstash

ghehe, just had a redis backlog of 300k items... adding a 3rd LS node and set the number of workers to 10 took care of that

logstash is awesome :)

Lorax, https://paste.as5580.net/p/gsjund2gbtyuahr4

jamesturnbull: I must apologize, after ripping the book out of the kindle reader into a pdf I can actually see whole chapters.

Lorax, the config is from etc/logstash/syslog.conf

the input I am using for syslog is - https://paste.as5580.net/p/ttuvdempo8iej31t

Lorax, the following is the config at the agent side (rsyslog confs)

avail in /etc/rsyslog.conf

Hi all

I still have the same problem for my timestamp and i tried this config file

Title: Try to grok - Pastebin.com (at pastebin.com)

Is someone know what's wrong with it ?

New news from newjiraissues: Jan Stenvall created LOGSTASH-1253 - Lumberjack connections stay in CLOSE_WAIT state indefinitely <https://logstash.jira.com/browse/LOGSTASH-1253>;

hi everybody, what's your opinion on ELMA? http://code.google.com/p/enterprise-log-managem...

<http://goo.gl/SDjiTp>; (at code.google.com)

Lorax,are you there?

Lorax, The last 10 lines were added to the logstash confs

Hi all, I get this error while starting logstash jar (compiled with PR https://github.com/logstash/logstash/pull/673 ). http://pb.abhijeetr.com/VSfE Any idea as to what might be wrong. It "works on my local laptop" but not on a server I've. What can cause this?

Title: Fix prefix and include exclude keys related bug by shadyabhi · Pull Request #673 · logstash/logstash · GitHub (at github.com)

Can anyone give a clue as to what might be wrong?

So I notice every time the month rolls over, certain logs stop stashing.

For awhile.

They randomly come back

anyone else have this?

hi, I'm trying to parse httpd logs in Common Log format, not Combined Log

SMALLHTTPD %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

somehow this gives me parse failures or no output in logstash, but it passes the rspec tests fine

kamenote: try it in the http://grokdebug.herokuapp.com/

kamenote: can you pastebin an example log line? and indeed try it in that app ^^

Is anyone using beaver? Im trying to figure out using add_field configuration

Does it have to be a hash or an array?

hm, how can I count a TOP10 for IP address hits in Kibana?

thanks lieter and electrical, I'll try it there :)

burn, terms panel

fwiw, I just needed to reload kibana :/

IT4Women: sorry, at work, IRC is at least secondary. :)

lieter: I see, and how to count a top10 of a certain value?

now I'm getting the amount of log lines

Lorax, I see

adepasquale: opensuse instead of debian? feh!

burn: you can fill in the field you want and select the quer{y,ies} you want to see the terms for

lieter: that's the problem, I only can find elasticsearch docs/examples, not for Kibana

IT4Women: did you see my link with the explicit tcp/udp config?

Lorax: i'm a gentoo user :P what i'd like to know is, do you like the idea?

does anyone happem to know what the minimum space requirements for the shipper is? I need this for the documentation I am writing...

Lorax, I saw it. I replaced the codes to the existing one

adepasquale: I don't like appliances, that could be done in a rpm or deb and not suffer from the package lag that results in 0day exploits.

Lorax: plse look at the logstash confs and tell me. the ones I added are the last 10 lines

for example, terradata gave us a PoC appliance to attach to a hadoop cluster, it took metasploit about six seconds to open up three different shells.

lieter: it seems I need to use facets?

Lorax: do you also think redis is better?

lieter: nvm, found it

IT4Women: if that's the whole file pasted in, you commented out the close to the output { }

I want to debug an issue I'm having with the file input. I have the src, and I want to modify the code or dep gem to add more logging, then I want to run the app. What's the best way to run it? Ideally I don't have to generate the jar, and I can run it from java.

Lorax: one of their points is also that compressed rsyslog takes way less space than compressed ES. what's your opinion? http://code.google.com/p/enterprise-log-managem...

<http://goo.gl/jvn1oa>; (at code.google.com)

If I have to do that, I need a way to get my changes in the deps that get built into the jar

carlobarbara: eclipse? :)

adepasquale: well of course it does, it's not parsed

I'd like to see some search times benchmarked

uncompress and read through vs check index, read count.

Lorax: i'm sure you'll laugh when you'll see that numbers :)

thanks for your opinion, I appreciate that

Lorax: I've developed in java and ruby, but never jruby. So I'm just trying to figure the workflow for making a change, building, and running. I know I can use eclipse to edit files & run java, but that doesn't tell me how to build my changes

nvm, I think I can do it this way

bin/logstash agent [options]