#logstash

jerrac: If HTTPDATE is already extracted as a field, then i think you'd do: match => [ "HTTPDATE", "MMM dd YYYY HH:mm:ss", "MMM d YYY....]

terroNZ: I think that's what I'm doing with that line I pasted above. Not sure of the performance impact, but we'll see soon enough :)

walkeran: it gets extracted by grok. So, set the data filter after the grok filter?

jerrac: Ah, actually... it extracts that ( https://github.com/elasticsearch/logstash/blob/... ) as the field "timestamp"... so if you put the date filter after the grok filter, you should be good to go

<http://tinyurl.com/megs9vk>; (at github.com)

As long as you match the format correctly

I'd do: if [myfield] { mutate { convert=> ["myfield", "string"] } }

thanks, thats what I thought. :)

terroNZ: Oh, thanks! I don't know how I missed the convert setting :P

would I use the mutate filter to override the host field?

jerrac: Yeah, that should do the trick.. either add_field to add a new host field, or replace to replace the current one

thought so. :)

i dont suppose anyone is aware of a grok pattern for python tracebacks? I've been googling around. I guess all you'd need to do is match the first line "Traceback (most recent call last):" and then every indented line after that and finally the first unindented line after those all belong together?

Does the puppet-logstash module have a setting for java memory usage?

idiot kibana question … how do you lock a colour to a specific term ( e.g. I’m doing searches for types and also a pie chart and the ‘api_access’ type is a different color in each

walkeran: FYI, it worked perfect. Thanks for the tips! :)

is i possible to cluster 2 logstash servers?

is it possible to cluster 2 logstash servers?

thumpba__: cluster? - what do you mean

Has anyone used any contrib plugin with logstash 1.4.0?

if i needed to create a pattern for a log entry that is always in the exact location but would sometimes be something.7364.something: or something.something how could i do it with the same syntax name?

I have isuru

freezey: something\.(\d+.)?something

freezey: something\.(\d+\.)?something

yeah i got it

wondering how how i can strip the numbers in the middle

and make that its own item

or tag

probably using named capturing groups

haven't done it in LS

Thanks terroNZ

I'm having a issue https://groups.google.com/d/msg/logstash-users/...

<http://tinyurl.com/m24r8f4>; (at groups.google.com)

LoadError: no such file to load -- jmx4r

when trying to use JMX input plugin

how did you install the plugin?

with plugin install?

./plugin install contrib

yes

what is -p /Users/sroy2/logstash/logstash-contrib-1.4.0/lib supposed to do?

I'm trying the command ./bin/logstash -f ~/temp/logstash-conf/conf/test-jmx.conf -p bin/lib/

try without -p

just ./bin/logstash -f ~/temp/logstash-conf/conf/test-jmx.conf

Is it recommended to have a redis queue or something between LS and ES, in case it needs to buffer? Or will LS handle that sort of thing fine? Or will it be ls-forwarder that does the buffering/waiting if needed?

bryanp: I've usually seen a queue between inputs and processor nodes, with processors feeding directly to ES

Hmm… logstash-forwarder sends directly to LS. So you’re talking like… A separate LS server that simply takes those, doesn’t process and dumps them to redis? Then my processor nodes pull from redis?

without -p, I get "Couldn't find any input plugin named 'jmx'."

that is really weird

copy the plugin to the correct folder and try again

just untar the contrib tarball in the same directory LS is installed

yeah.. this is weird.. trying this many times in 1.4.9

you should end up with every contrib plugin in the correct directory

*1.4.0

it works with 1.4.1

that's how I did it

just untar and go

okay...

let me try that

Whats the recommended LS_HEAP_SIZE?

terroNZ, it worked!!

Thanks a lot terroNZ!

no prob isuru

bryanp: Ah, didn't realize you're using the forwarder. There was some discussion about redis support, but a quick skim suggests it's unlikely https://github.com/elasticsearch/logstash-forwa...

<http://tinyurl.com/k2fj8tn>; (at github.com)

rutter: No worries. I’m going to just wing it and see if my new ELK cluster can keep up fine. Should be able to

Is 2GB memory enough for LS? It’s on the same box as ES, trying to figure the right balance.

Ugh, looks like the flume exporter isn't actually logstash compatible? It's confusing kibana. My mapping file looks like:

<http://tinyurl.com/q9x3fm7>; (at gist.github.com)

can somebody paste their mapping file in elastic search to compare - the timestamp as text seems iffy

cschneid: here’s what i’ve got. Not sure if it’s right tho :P https://gist.github.com/bryanpearson/166e99af08...

<http://tinyurl.com/lwq76tp>; (at gist.github.com)

bryanp: yeah, I'm getting a kibana error w/ the @timestamp thing

ahh well, this is a problem for another day. Data is landing in flat files, which is plenty - and I'm out of energy for doing devopsy things today. Thanks for that link - I'll probably be back asking later

might end up just using logstash proper... if it works

bryanp: re: heap - really depends what's happening

imperialwicket: k. I’ve got two 12GB VM’s. 5gb for ES, and 3GB for LS on each. Sound reasonable?

imperialwicket: ~1mil events/day

depends on what you need them to do, i'd expect that to handle plenty though

imperialwicket: Ya, I’ll give it a try. That’s more than the single VM has now, and it’s handling it fine. But it’s got a redis buffer if it needs, this new setup wont