#logstash

anyone knows if topN features support regexp ?

electrical: Maybe you should do nightly builds for the brave ;)

rgj: could do that yeah

electrical, if I start logstash with a debug I see no errors in log, however if I start with -t, I see the config test end with an OK, either way the java process dies shortly after...

I'm thinking it may be a open files issue with es on same host perhaps, have you seen this before?

electrical: it fails on some ruby gem deps in vendor/bundle

rgj, electrical: +1 on nightly builds

electrical: if you could build it for me it would be awesome :D

anyone using multiline with postfix mail.log output?

and hav a good regex for pattern

multiline with postfix?

hmm.. wasn't aware postfix ever logged multiline

yea its 3-5 lines for an event

I have a good Postfix grok pattern set for single-line entries

mine isn't... :|

odd

3-5 lines or just wrapped in your terminal?

cat /var/log/mail.log |grep 759FF4B87E88 |wc -l

5

for example

:(

poolski what kind of zimbra do you use ? (community or network )

that might just refer to that QID showing up 5 times... but ok

network

we use it for our company mail

how many users ?

70ish mailboxen

ok

and hardware setup ?

all our stuff's virtualised - the zimbra box has 6(I think) CPUs, 24G RAM

I'm looking forward to ZCS8.5

more distributed/modular infrastructure for better redundancy/expansion

poolski:

Apr 1 08:30:08 artilabs postfix/pickup[16942]: 759FF4B87E88: uid=2000 from=<user>

Apr 1 08:30:08 artilabs postfix/cleanup[18892]: 759FF4B87E88: message-id=<869cbd6d6331ea7242c0ef01af764a90...;

Apr 1 08:30:08 artilabs postfix/qmgr[21380]: 759FF4B87E88: from=<user@domain.com>;, size=924, nrcpt=1 (queue active)

Apr 1 08:30:10 artilabs postfix/smtp[18894]: 759FF4B87E88: to=<christian@user.se>;, relay=smtp-domain.com[192.168.5.10]:25, delay=2.4, delays=0.03/0.02/0.13/2.2, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as D2E1D15E1342)

here is an example

yeah

that's 5 lines

not one multiline

I would like to merge these into one line

so you don't need a multiline filter

aaaah

you mean one "event", I suspect

grouped by QID

yes

?

searching these is not that easy in kibana as you cant say show +-2sec

no, but you CAN search by 'from' or 'to'

unless you want to see the entire lifespan of the message

I want to see the entire

is this a pure postfix box or a Zimbra install or something?

for example opsfp blocks etc doesnt show otherwise

ospf.

cause trying to do that with Zimbra is a hopeless affair, since each message goes through like 5 queues and generates 5 QIDs :(

it makes me cry

poolski: pure postfix

let me dump my postfix grok into a Gist

thanks but is there a filter to merge lines based on regex that matches?

Title: Usefuk Logstash GROK patterns (at gist.github.com)

never considered that, thanks for the idea ;)

looking at the docs, it doesn't look like you can -

however, you could search by QID in Kibana and get the lifespan of the event

it's almost better to have 5 events as you can see what was happening to your message at ever step of the process

http://logstash.net/docs/1.4.0/filters/collate this seems to come close, but I've never used it

Title: logstash - open source log management (at logstash.net)

electrical: here´s the error message when trying to make the tarball from github master: => Ensuring ruby gems dependencies are in vendor/bundle...

make: *** [vendor/bundle] Error 1

poolski: I'm gonna steal your amavis stuff :P

Who is maintaining the relp-plugin?

eritho: ah.. did you do 'make clean' before?

electrical: same error after running make clean and then make tarball

lieter, go for it

why does lostash 1.4 tries to start logstash-web every second on my agents?

also, does setting prefetch_count to 0 stop LS from prefetching from RMQ?

I need it to handle one event at a time

Hi, anyone can help me? Having issues with removing fields using logstash. Current configuration nxlog (gelf output)-->logstash (gelf output)-->graylog2. Using the filter mutate but it doesn´t work, the field that needs to be removed still appears in Graylog2

filter { mutate { type => all remove_field => [ "version", "Keywords", "facility", "type", "full_message" ] } }

eritho: okay. you might need to remove the vendor dir

Hmmm ..

Cyis|afk|afk, you astill AFK?

electrical: same error after removing vendor dir, make clean, make tarball

hmm. let me check on my dev machine

k.. thnx for the help btw..

np :-)

poolski: yes that is what I do today, first serch for email and time and the grab QID

and do a second search

What is the easiest way of getting rid of the portnumber in the host-field? Do i really need to grok it?

but mail from and to are not in the same line

if I'm not mistaken

so can be hard to find the correct one

if you have a heavy trafficed mail cluster

poolski: https://github.com/elasticsearch/kibana/issues/275 this issue is probably a by effect of not beeing able to group logs into a single event

<http://tinyurl.com/pbzckhl>; (at github.com)

Ha! For purely non-profit reasons, all your nickserv accounts have been converted into freenode+ accounts;

habanero_: hehe yeah

i don't believe it

afterall. its April 1st

it's 6 minutes late for april fools -but it must be a joke

Late? It's still April 1st here :)

i thought there was an (unwritten) rule about the joke having to be before midday...

nah :-) they can do it all day :p

it's a good'n' anyway

Hi, anyone can help me? Having issues with removing fields using logstash. Current configuration nxlog (gelf output)-->logstash (gelf output)-->graylog2. Using the filter mutate but it doesn´t work, the field that needs to be removed still appears in Graylog2

And in the states, some of them are still sleeping.

and there´s not much documentation on how to troubeshoot it :-(

bruun_: Maybe a link to your config and a description of which field that's not being removed.

thanks rgj, placed post on google , link : https://groups.google.com/forum/#!topic/logstas...

<http://tinyurl.com/4kcmbx7>; (at groups.google.com)

really appriciated, because i don´t know where to look anymore after troubleshooting for hours :-)

is there a way to throttle how many messages LS tries to grab from RMQ?

I have a 300k event backlog and it looks like LS is trying to grab FAR too many at a time

bruun_: Not sure that you can do type => all .. Which anyway is deprecated way of configuring logstash.

poolski try to to divide the queues