#logstash

via logstash, yes

BaM, i like that idea better than mine

it's also scalable with hardly any effort

i was using output:file after sending to logstash from remote rsyslog

its just not the same

rgr

since each bit pulls fromt he queue, you just add more workers

tm23: I only just installed it within the last three days let me have a look

there'a a bit of vrrp in there to make it work from the filter cluster back to the queue

i understand receiving with syslog and saving to local files, but to also send to logstash do you just add a basic *.* @@ localhost:514 ?

why would you send to a localhost logstash?

or whatever port logstash is listening on

just have it pick up the files you want

chandler1: @@ for tcp, @ for udp IIRC

yeah

tm23: lumberjack-0.1.2-1.x86_64

chandler1: maybe take a look at http://www.rsyslog.com/doc/rsyslog_tls.html

i thought you were using local syslog to 1) save a local copy of whatever logs were received, and 2) also forward them to the local instance of logstash for analytics

shaynem_: huh, I'm running 0.1.0 on centos 5.9

shaynem_: lemme find another 5.x box :)

you got a rpm :P

I do

Actually, my test log spewer is a 5.5 box

The Go website specifically says it doesn't support Centos 5.x

even tried older versions of go

http://golang.org/doc/install --- Linux 2.6.23 or later with glibcamd64, 386, armCentOS/RHEL 5.x not supported; no binary distribution for ARM yet

Title: Getting Started - The Go Programming Language (at golang.org)

shaynem_: huh, I guess he bundled in enough go to get it to build in 0.1.0 :)

really need to stop using limechat on osx - irc client just crashed

I'll try an older version and let you know

is logstash 1.2 available? is it integrated into kibana 3...do i just install kibana 3 and it will setup logstash 1.2 ?

i'd like to use the kv filter, but only whitelist certain field names.. alternatively i could write a grok filter per whitelisted field name, but that seems like it would be slower

hm getting errors on my input, is it possible to list 3 tcp inputs with different ports?

shaynem_: I built the rpm on a 6.x box and it seems to work on the 5.5 box

tm23: so it's just me then

I'm just building the older version now

https://logstash.jira.com/browse/LOGSTASH-663 - 'kibana 3 is now bundled with logstash builds as of 1.2.0.dev' so i grab the master branch of logstash to get kibana3 aswell? (confused)

Title: [LOGSTASH-663] More visualizations for Kibana - logstash.jira.com (at logstash.jira.com)

tm23: Did you build 0.1.2 or 0.1.0 ?

0.1.2

:/

let me try on a different box

grepusername: you don't have to grab LS to get kibana, but if you're grabbing a 1.2 LS, you get kibana inside the jar

also, you don't have to grab master, there's 1.2.0.beta

beta1*

I should update kibana 2 for the new schema

tomorow

tm23: worked didn't seg fault - I guess it's something wrong with the box i was testing on.. however Thank you for this :-) otherwise i would have ruled it out.

shaynem_: np

shaynem_: thanks for bringing it up, otherwise, I wouldn't know to build lumberjack on a 6.x box :)

:P

when you guys are making a new grok pattern, what do you use for reference? the original log? when it's transferred extra info is added, but using logstash to output to stdout all i see is the result post-logstash

how are you guys viewing the events post-transfer before logstash parses em

BaM`: pong?

hey

how's it?

I have a question from the other day about the multiline filter

law: pong

whack: check out the stupid amavisd splitting example at http://bugs.debian.org/cgi-bin/bugreport.cgi?bu...

<http://goo.gl/jkqhTA>; (at bugs.debian.org)

can multiline deal with that sort of thing?

simmel: yeah I'm planning on having an RC available tonight; just gotta hack on a few small things

BaM`: will take a peak in a sec

I really want to strip the second lot of syslog stuff off the start of it

kk

BaM`: I feel that log message was too much anonymized, so I can't say for sure

but yes I believe the multiline support in logstash should be able to handle that

I really want to remove the ... from the tail end of the first line, and up to the ... in the second line

problem is everything before the second lot of ... is going to differ per message

hm. doing a mutate removal on @message leaves behind "json_event" and removing @source also seems to nuke the @source_path

BaM`: can you show me like 2 or 3 whole messages? (replace any emails with 'example@example.com'; or something)

tm23: not sure I Understand the first part

sure - one sec

is it possible to use the mutate filter to update "@type"? i was using grok to add a "type" field but i'd like to update the top-level value. i tried using mutate. didn't have any luck.

but the 2nd part (removing @source causes @source_path to disappear) seems like something I wouldn't expect

zounese: yes!

zounese: mutate { replace => [ "@type", "%{type}" ] }

reads the value of the 'type' field and stores it in @type

hmm. i tried that. perhaps i made a typo.

thanks for confirming.

whack: mutate { remove => ["@message", "@source"] } in kibana3 @message shows up as a field with the value "json_event". maybe an ES weirdness

tm23: unrelated

tm23: I bet your input sets "message_format => json_event"

message_format sets the text value for the @message field

whack: ahh

long term

message_Format is going away

too confusing ;)

not super useful