#logstash

by default, kibana would retrieve last 15 mins

yeah

I could see elasticsearch indexes for today is being updated with current timestamp.

what else I need to check?

jameshyde: try widening your search period?

have you changed the address of ES?

whack: I tried 'all' already, no luck

hey whack :)

devOpsEv: it is localhost / 127.0.0.1, so no

could you help me quikly ...

is it possible to use GROK to pull all of the @ from the folllowing ... " Tweet stuff ..... @name @name @name"

but i dont know how many @ there will be ..

there could be 1 there could be 10 ?

i cant seem to find a way with grok ...

btw kibana 3 is awesome .. add some alerting capability and i would pay for it !

logger72: check out the workaround for modsectag in ,,(mod security)

logger72: I do not know about 'mod security', but I do know about these similar topics: 'modsecurity'

logger72: ,,(modsecurity)

logger72: mod_security crs log parser: https://gist.github.com/1346387

s/modsectag/MODSECRULETAGS/

captures up to 10 things

allows for more though they're not captured

ah .. nice ..

thats just the ticket ..

sorry power ran out :)

btw .. what do you reckon the vm spec would be etc Xm settings for 12Mbyts per second of log traffic < 512k in size ... ?

<512k per event i mean

i have SSD so i/o is ok ..

also i didnt realise that ES was tuned for read rather than write ?

question regarding multiline support. I currently have my multiline{} filter rules in the shipper's config. I'd love to be able to move as much custom config/filtering/grok as possible to the central server's config. So, the question is: if multiline messages are broken apart by the shipper's config, can they be re-assembled by similar rules in the central server's config?

i changed it and it made a massive difference on iops

amazing, someone forked & updated my modsecurity crs parser! https://gist.github.com/schewara/5668621

Title: Logstash parser for ModSecurity/CRS entries in the Apache ErrorLog (at gist.github.com)

semi .. did you see my windows GROK parser ..

did not

does the full windows evt log ..

i'm sorry to hear that ;)

shall i pasty it for you

you want it ?

no thanks

not into windows

lol .. me neither .. i had to do it for work :)

although if you want to share with others, put it in a gist and teach logstashbot the link

how do i teach ?

`learn foo as bar

semiosis: OK.

`foo

semiosis: bar

`forget foo

semiosis: OK.

`foo

semiosis: Error: "foo" is not a valid command.

k

thats all you need to know

you could also i suppose contributed to the logstash cookbook

i tried , i get a 404

oh

its been like that for ages ..

i have snort, windows , cisco asa

oh an brocade stuff as well

i see you take your handle seriously

eh ?

logger72, you deal with logs a lot

yeah i do ... i also have twitter :)

EEAA: that should work

thats interesting . .i hacked a smiley face into Kibana .. when there are a load of :( the face looks glum and vice versa ...

EEAA: according to the docs, multiline uses the event source as well as a regex pattern to assemble events from multiline, so should be fine

devOpsEv: ok, excellent. I'm just sick of having to maintain agent-based config. I just want to have the agents suck up log and then I'll deal with organizing it all in the central server.

I hear ya

My shipper configs are like 15 lines at most

devOpsEv: I'll hopefully be there soonish.

i just spit them all onto a message queue ...

input { some files here } output {redis} and that's about it

probs with that is though the are unordered ...

logger72: good point. Dangit.

logger72: our devs *really* don't like trying to manually sort unordered python stack traces. :)

what do you mean unordered? not chronologically ordered?

the message queue is not pop/push its random

devOpsEv: yes. Applying the multiline filter on the shipper solves that problem.

logger72: hrm, is there any way to make redis FIFO?

redis is pop/push though

yup ..

redis is FIFO by default

ok

i have to use our enterprise bus which isnt :( which is a bit shit for me ...

just random? not stacked?

then I wonder where the un-ordering is happening. Cause we saw this exact thing before I implemented multiline filtering.

redis is FIFO by default ?

sweet

at the moment, I only have a single redis consumer box, so that wouldn't be the source...

actually

it's... not

but

i didnt think it was ..

logstash treats it as such

you can lpop or rpop

so you can pop from either end or push from either end

yes but the subscriber may not get events in the same order they went in ..

logstash always does an lpush on the output, and blpop on the redis input

k

devOpsEv: rpush

btw how do you escape @ in grok ?

right, right

so it does treat it as fifo

logger72: I don't understand the question. You don't need to escape '@'

kk

so newest events get pushed to the right end of the stack, and oldest events get blpop from the left end of the stack

so redis itself isn't fifo or lifo, depends on what redis commands you use

logstash treats it is fifo

s/is/as

so if I have python.log -> logstash-shipper -> redis -> logstash-central -> elasticsearch, why would multiline logs come in out-of-order (without using multiline filter on the shipper)

EEAA: they should come in order. afaik

I always run multiline on the shipper, so it doesn't have this problem.

EEAA: can you show an example of this out-of-order-ness ?

whack: I can, once I get my ES out of the funk it's currently in.

gota drop off for a sec...

morning (somewhere it must be morning)

michaelk: cpu utilizaton for the logstash that's ... well it's not exactly hanging but just going really slow ... is 0%.

wb JoeJulian