#logstash

Knuit: wow, the QS regexp is highly unreadable :(

BongoEADGC61 I found that link, but what is the --configtest input flag? Is it --configtest or --testconfig?

soulair: /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d --configtest

--configtest

ahh there we go

much obliged

Any time

soulair: ./logstash --help for the next time you forget :)

rastro: works fine with %{DATA} in the debugger where %{QS} does not, testing it in LS now

Knuit: i would hope it would fail in LS, too.

It does in fail with %{QS} in LS, yeah

rastro that rocks!

Knuit: i love regular expressions, but the %QS is way too weird for me.

Is there any good article or documentation on how Logstash combines multiple config files

rastro…ok I got it

it was the version difference

in my experience it works kinda like cat * > conf

I am using alphanumeric patterns on my file name to sort... ie 40<name>.conf goes befor 60<name>.conf

once I changed to 1.4.2 it works

thanks for your help

I just need to make sure that we have current version

with Es

Bo7a... That was my experience too but I am seeing weird behavior with 1.5RC3 and I can't seem to lock it down...

has anyone ever seen a stat.st_dev error when using the file input?

I am up to 2468 lines in my config so dividing and symlinking is really important for clarity sake. Was there any changes to how it is config is combined?

this is the error stat.st_dev error while using the file input.

Title: NotImplementedError: stat.st_dev unsupported or native support failed to load - Pastebin.com (at pastebin.com)

Hello, How to know How many events a Logstash server Processed Per minute.

what user does the logstash run under?

asimzaidi: the logstash user.

asimzaidi: At least if you install the Debian packages. Else as whatever user you start it with.

logstash send my logs from /tmp/logstash.txt but not from /var/log/apache/error.log

thehybridtech: what behavior are you seeing?

nothing ..it just doesnt do anything

mpnoob: "it depends"...

mpnoob: LS is more cpu-intensive, but it relies on the availability of your ES cluster...

mpnoob: there

mpnoob: there's a config here that's supposed to measure your throughput in LS: http://logstash.net/docs/1.4.0/filters/metrics

Title: logstash - open source log management (at logstash.net)

rastro... Not exactly sure how to describe it. I have a gsub that is puts "SPLITLINE" in sections that I need to split at. I then run split later in the config and it works. This was working in RC2. If I move the split pattern later in the process (Another file) it works... SPLITLINE shows up in the sections that were supposed to be split so I know GSUB worked at one point.

thehybridtech: is the first gsub conditional on anything that maybe doesn't exist yet?

rastro: no. Log I am parsing has epochstamp with a timestamp following after the controller loads up... I basically do a multiline on file input to put all the epochstamps with the timestamp; split the epoch into multiple sections (Config/State/Load Logs). The gsub handles that function by adding SPLITLINE. I then SPLIT again on every line once the data I need has been extracted from each section. The split I am having

problems with is the first split following the SPLITLINE add.

JAS... Let me add a basic config on pastebin

I had a typo in my grok line. Is there a possibility to reparse all entries that are tagged "_grokparsefailure"?

petn-randall: i haven't done it, but you could select them with the elasticsearch{} input.

rastro: I see, but how do I "trigger" it? Or does it continuously poll from ES?

petn-randall: as an input, it will run the query against ES.

petn-randall: each document it finds should become an event for LS to process.

petn-randall: when you output, use the document's original ID and it will update it. Or, let it go with a new ID and delete the old ones when you're done.

rastro: So the actual query against ES will only be done once at start, right?

rastro: Because if ES pushes new results continuously, I could easily end up with an endless loop, which I'd like to avoid.

petn-randall: not so sure, because there is a 'size' param that defaults to 1000 records to process.

When using stdin as an input, would i just pipe output of a cat command into logstash in /opt/logstash/bin/logstash?

petn-randall: your new records shouldn't have the GPF, and, if you update, the old ones won't either.

soulair: or paste to stdin, either one.

sweet

rastro: I also see the scroll parameter, which is some sort of timer.

well my agent is running as a service, does that affect anything?

effect*

soulair: yes, it means you don't have a stdin. run it from the command line.

petn-randall: are the old records worth the hassle? Can you delete them and reimport them?

rastro: "could". I could drop the records, and push them via rsylog again. But I'm guessing just reparsing is easier.

rastro: thanks, im doing that now. you didnt happen to see that error I posted earlier?

soulair: don't think so.

rastro: so it is when I am using a file input. its happened on two different ELK configurations now. here is the error pastebin. http://pastebin.com/LSER3eax

Title: NotImplementedError: stat.st_dev unsupported or native support failed to load - Pastebin.com (at pastebin.com)

rastro: it's also being discussed here on github: https://github.com/elastic/logstash/issues/3033

Title: Using logstash 1.5.0 we receive stat.st_dev error when reading from file · Issue #3033 · elastic/logstash · GitHub (at github.com)

soulair: also here: https://logstash.jira.com/browse/LOGSTASH-665

soulair: see jordan's comment from 3/7/13.

sorry rastro, i dont exactly know how to do what he is saying to do.

soulair: but maybe that's not it. the thread ends by blaming ruby.

i think it is a JRuby issue

b/c there is an issue on JRuby github that people are suspecting is related

oh so I saw that issue recently

now I saw it on Solaris

moment