#logstash

http://pastebin.com/WsiccmAz is what I see on startup, which is nice as they all seem to be easy to fix

Title: [Rails] Using milestone 2 input plugin 'tcp'. This plugin should be stable, but if you s - Pastebin.com (at pastebin.com)

Is there a way to write a filter that only applies to messages in a specific datetime range?

mfn: good luck ;)

I would have to write it as a regex, right?

is "pattern" depreciated meaning they need to live outside the conf?

Lorax: no just use match instead

kossmoboleat: there's also a plugin which allows to execute ruby code .. you get passed in the "event" ... not sure if that approach would be better though

s/better/better than regex/

hmm, I don't know ruby yet, so maybe not...

kossmoboleat: http://pastebin.com/B8Q7hb0S would I replace that with a match => ["message" => the_syslog_pattern ] ?

Title: Oldstyle Logstash pattern match. - Pastebin.com (at pastebin.com)

Lorax: yes, that's the idea

instead of writing type =>..., you'd write if [type] == "syslog" { grok { ...

do you know if _grokparsefailure is still available? I'm using an almost verbatim copy of the syslog parser from the cookbook http://cookbook.logstash.net/recipes/syslog-pri/

Title: parsing syslog messages - logstash cookbook (at cookbook.logstash.net)

centra-syslog also has the older style grok

Silly question, Grok Filters, I want to filter [eventnumber] do i need to escape the square brackets?

gme: yes, prepend a \

mfn: cheers

gme: I learned by looking at the predefined grok patterns: https://raw.github.com/logstash/logstash/master...

<http://goo.gl/PcQxmS>; (at raw.github.com)

New news from newjiraissues: David G created LOGSTASH-1335 - DNS lookup error when using elasticsearch_http <https://logstash.jira.com/browse/LOGSTASH-1335>;

How long does Logstash store logs in Elasticsearch? Does it periodically delete old indexes?

it never deletes anything

I'm trying to search on only one indice (todays logstash..) http://pastebin.com/bcsZX7Md - but it fails miserably.. :(

Title: { "indices": { "indices": ["logstash-2013.09.30"], "facets": { - Pastebin.com (at pastebin.com)

you can do things like - https://github.com/logstash/expire-logs to delete old 'logs'

Title: logstash/expire-logs · GitHub (at github.com)

eper: Thanks, I'll check that out.

I love it, python to maintain ruby running in a JVM.

logstashbot seen mtds

secti: mtds was last seen in #logstash 2 days, 21 hours, 36 minutes, and 52 seconds ago: <mtds> gotta go. See you folks and have a nice weekend!

logstashbot seen devOpsEv

secti: devOpsEv was last seen in #logstash 2 days, 16 hours, 36 minutes, and 4 seconds ago: <devOpsEv> and with that, so begins my weekend. o7

FiXion: You can just send your query to http://localhost:9200/logstash-2013.09.30/_search/ to limit your search to that index

Hey guys, I'm trying to pass logs from a queue to elasticsearch, the message fromt he queue is in json format, part of the message is a nested json object and this seems to be what's throwing the problem. https://gist.github.com/gabriel403/d34ee7107cca... this contains the error log message from es being recorded by ls and the json it's failing on

<http://goo.gl/mBxalK>; (at gist.github.com)

xorpaul: thanks

so how do I increase the heap size or whatever's needed to avoid java.lang.OutOfMemoryError: GC overhead limit exceeded ?

faxm0dem: I think you want the -Xmx option for the java command.

xorpaul: Did that log I posted earlier tell you anything?

GregMefford: thanks

Anyone hangin around?

trying to get PHP logs working with logstash. I am using filters found here: https://logstash.jira.com/browse/LOGSTASH-770 and it comes with _jsonparsefailure

Title: [LOGSTASH-770] Doubled @fields values - logstash.jira.com (at logstash.jira.com)

is there a standard set of grok patterns for PHP logs?

I'm new here... I'm guessing you have already looked at this? http://grokdebug.herokuapp.com/patterns#

Title: Grok Debugger (at grokdebug.herokuapp.com)

I've managed to get a Fortinet/Eventlogs working... It was alot of trial an error, but it worked.

grzany: The built-in patterns are listed here (https://github.com/logstash/logstash/tree/v1.2....) It doesn't look like PHP is in there.

<http://goo.gl/fVGjhp>; (at github.com)

Did you look through this? https://botbot.me/freenode/logstash/search/?q=php

Title: Logs for #logstash | BotBot.me [o__o] (at botbot.me)

GregMefford: yes, I can't see. But surely someone has came up with patterns for PHP logs - it's so common

Sorry firewall died, don't suppose anyone had any inkling as what I need to do to allow nested messages in es?

grzany: The other place to look is here (http://cookbook.logstash.net/) but I don't see it there either. I'm sure people have implemented it, but they haven't shared.

Title: the logstash community cookbook - logstash cookbook (at cookbook.logstash.net)

oddur: Not really, I don't understand that you don't get anything even from the stdout plugin.

gme: I just looked at the GC activity of my LS instance and the JVM settings "-Xmx1024m -Xmn512m -XX:TargetSurvivorRatio=90" helped too keep the New GCs down from 2 every second to 1 every ~40 seconds, but still not really noticable I think.

SonOfCar: can you add your conf to the cookbook? that would be useful for me as well

I get a "404" error when I try to edit the cookbook. https://github.com/logstash/cookbook/edit/gh-pa...

<http://goo.gl/wQplHU>; (at github.com)

xorpaul: Alright. As far as I know I've opened all the necessary ports on the machine, and if I manually POST something to ES then that data gets created just fine and at least Kibana doesn't complain about no indices existing. So, I'm kind of stumped.

that's unfortunate, perhaps that's why it's lagging behind

anyone can help to configure an agent to send logs to logstash using rsyslog?

AlBundy: this really helped a lot, 1.2.0 is now running very stable in production as shipper, thanks

(as stable as something can run for half an hour :)

yw :)

mine crashed earlier this week because of open files

soso: Just use the syslog input plugin

i followed instructions from internet but it is not shipping