#logstash

fetep / whack: PR #423 can be merged.

is putting a small config snippet into the header ( i guess you call it ) for the docs ok? I understand why the csv doc lists a regexp as one of the options, but it's not exactly obvious what it means when you're trying to make the thing work

yay

again concerning LOGSTASH-310 I found this, https://github.com/brightcode/smtpd/blob/master...

<http://goo.gl/HjKwe>; (at github.com)

Jira issue [LOGSTASH-310] Mail input - logstash.jira.com - https://logstash.jira.com/browse/LOGSTASH-310

is there anything specific to logstash when considering disk space requirements?

I imported a 5.9MB file and ended up with a 40MB data directory.

blalor: logstash really doesn't use much disk at all. are you asking about elasticsearch?

well that's elasticsearch, not logstash, technically

sorry, yes, I was THINKING elasticsearch, failed to TYPE it.

geez, you can't read a newbie's mind. that's a bug.

the rule of thumb is you'll need ~3x raw log size on disk, if you don't do anything special like enable compression or mutate logs to reduce redundancy

the 3x probably varies widely though

blalor: elasticsearch inflates static log data because a bunch of meta-data is being generated to go along w/ it.

blalor: couple pieces of advice: make sure you have the compression stuff on

semiosis: thought it was 6x ? ( disk tests from whack )

blalor: file it in ,,(jira) please :P

blalor: https://logstash.jira.com/secure/Dashboard.jspa

heh

blalor: and you can experiment with index templates, so you don't do full indexing on @message (assuming you're grokking it out to @fields or otherwise have that populated)

Electrical: i'm sure you're right, my info is old

Electrical: it really depends how you've configured ES, what fields you're stripping, etc.

phrawzty: very true. but default config ( no stripping, no removing, no compression ) gives about 6x ( according to the tests whack did )

so far I'm just using the embedded instance as a demo, but sucking in about 770MB of prod logs from the last 4 months or so

gaah

i'm out of rubygas for the day i think

https://gist.github.com/timconradinc/5273375 <- i added a config snippit for the csv filter, i'm not sure it's the best place for something like that since other modules don't have it

Title: csv filter addition (at gist.github.com)

but it seems if other modules had it, it might help people too

i was gonna do a PR for it, but i can't make the docs to make sure it formats correctly atm

According to https://logstash.jira.com/browse/LOGSTASH-427 syslog will be dropped from logstash starting with 1.2.X. Should there not be a warning on the documentation page http://logstash.net/docs/1.1.9/inputs/syslog that mentions this?

Title: [LOGSTASH-427] Remove the syslog input - logstash.jira.com (at logstash.jira.com)

probably

it's still in head tho

or master/whatever

I've got timestamps in this format - "Mar 29 2013 14:19:52". The grok pattern SYSLOGTIMESTAMP won't match on that I don't think because the year is in there so I guess I have to create my own pattern?

ah, it's supposed to spit out a warning

techminer: https://github.com/logstash/logstash/tree/maste... <- might look through those

<http://goo.gl/RiQuG>; (at github.com)

mortini_: yea I've been looking at those….

yeah, most of it doesn't seem to follow that format

your format, i mean

I also have host entries like this "10.11.10.21/10.11.10.21" with the slash in the middle….and I don't think IPORHOST matches that either….

This pattern, if I formatted it correctly, should match my dates? %{MONTH} %{MONTHDAY} %{YEAR} %{TIME}

seems it should

i has no filterfu tho

….and this one my host IP entry… %{IPORHOST}/%{IPORHOST}

filterfu? meaning I can' t use my custom pattern to do anything?

no

heh

fu is like an understanding

or ability

or, foo, i guess?

when is the next version of logstash getting released ?...Is there any place where i can find it ?

I'm ok with fu

mortini_: I can assign my custom date pattern to a new value though and then use that?

kirankumar31: 1.1.9 doesn't cut it?

techminer: sorry, i don't really know, i've only been working with logstash for a few weeks and your questions are outside of what i've worked with

techminer: it should match your dates

techminer: you might try to grok match stuff on http://grokdebug.herokuapp.com/

Title: Grok Debugger (at grokdebug.herokuapp.com)

techminer: eg - insert input as "Mar 29 2013 14:19:52" and pattern as "%{MONTH} %{MONTHDAY} %{YEAR} %{TIME}"

(it'll match)

siert: I just found that too….awesome tool… thanks

techminer: and with 10.11.10.21/10.11.10.22 (note: changed 21 to 22) you can do something like

techminer: input - 10.11.10.21/10.11.10.22 ..... pattern: %{IPORHOST:source_ip}/%{IPORHOST:destination_ip}

siert: thanks

techminer: note the IPORHOST:<field_name> stuff (that's the way you create fields within logstash/kibana/etc)

yea I'm just now grasping this stuff….I'm trying to remove the first two fields…as the central loghost is adding an extra timestamp and hostname

techminer: syslog based stuff?

yes….rsyslog

techminer: you should read http://cookbook.logstash.net/recipes/syslog-pri/ and http://jpmens.net/2012/08/09/i-grok-how-to-muta... ...

<http://goo.gl/dv0OH>; (at jpmens.net)

techminer: it's quiet easy to create your own patterns as in 'MYSYSLOGLINE %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{HOSTORIP:syslog_hostname} %{GREEDYDATA:syslog_message}

New news from newjiraissues: Jordan Sissel created LOGSTASH-427 - Remove the 'syslog' input <https://logstash.jira.com/browse/LOGSTASH-427>;