#logstash

use date{}, set target to the field name you want to create

aaaaaawwww: so, you'll do this in 3 phases:

1) grok out the two timestamps into two separate fields

2) date{} for start time, set target accordingly

3) date{} for end time, set target accordingly

torrancew: thank you ... now how I can add %{DATE}:%{HAPROXYTIME} to a single field ?

aaaaaawwww: you can use a "native" regex concept called a named capture

(?<field_name_goes_here>%{DATE}:%{HAPROXYTIME})

basically, (?<field_name>PATTERN)

Thank you ... torrancew

ok whats wrong with this?

tag_on_failure => "_grokparsefailure_yealink"

and thm im trying to write all matching logs to a file using

<http://tinyurl.com/njlnyb7>; (at github.com)

looks fine offhand, TandyUK

ah i see, im getting it filtered by syslog by mistake

ah ffs i know why

oh why yealink do you force me to use port 514 for syslog

thats why i added my 'clienttags' filter

:)

how can i output the raw event using file { }

codecs, generally

yeah, plain?

codec => json will probably be the most sane

well

plain will dump the timestamp + "message" field IIRC

but you can override that

however, "json" will just serialize the event to json and dump that

yeah, im going for readable :P

data > input > fails to parse > output > raw log, as recieved, to file

TandyUK: then you'll have to muck with plain and its settings

ok

hmmm

yeah

so let's back up a sec

the ability to do that exists

but depends /heavily/ on how you mutate your events

apparantly plain is default for file

yeah

by default, will use plain

and will dump timestamp + message field

if that's enough, you can call it a day

but if you need to tune it, then the plain codec has some settings to help there

but requires knowledge of the fields you've already parsed

its ouputting json regardless lol

hmmmm

doc bug, maybe

manually set codec => plain ?

perhaps, done thats lol

how to output just "message" fro mthe json?

TandyUK: do you mean that you ahve "codec => plain" and you are still seeing json?

yes

k, one sec

(don't futz with that codec much myself)

<http://tinyurl.com/ntje3nq>; (at www.elastic.co)

oops, typo there at the end

anyway

try this: file { codec => plain { format => "%{@timestamp} %{message}" } }

forgot the path => but you get it

sorry, head is killing me today

yeah np :P

<http://tinyurl.com/njlnyb7>; (at github.com)

looks good

working?

actual log line pmed

no lol

10-4

TandyUK: what logstash version?

just upgraded to the latest 1.5

1.5.4-1 from the repo

cannot reproduce :/

cd /etc/logstash << might not work so well in here :P

haha

Title: Private Paste - Pastie (at pastie.org)

one of the few downsides of multi screens

that's me attempting a clean reproduction of this

cursor should type in the window im looking at dammit :P

(output buffering fail, btw)

haha

i'm like the 'mind controlled computer' interface designers worst nightmare

like now for example, id want to be able to run commands in one of the 4 ssh windows on screen 2, while still looking at irc :P

tiling window managers ftw

:)

Has anyone gotten the kafka input plugin attribute decode_class to work? When I use it, I get a class not found exception at run time, no matter where I put the class or the jar that contains is. Using logstash 1.5.4.

t4nk721: i've not tried, but let me peek at the code and see if i can help

t4nk721: have you mucked with the java env vars to add the jar to the classpath

yes, every way I try to modify it seems to be ignored.

including on the command line of logstash

interesting

https://github.com/tandyuk/ansible-logging-play... this should output to multiple files

<http://tinyurl.com/njlnyb7>; (at github.com)

agreed, it should

and rubydebug and json should be completely different right?

yeah, rubydebug is kinda like json, but multiline, and it's technically a ruby serialization, not a json one

ok, well diff thinks yealink-debug.log and yealink-raw.log are identical

o.O

seems not right

indeed

i don't use the file input much, and when I do its to generate json

got a meeting shortly, but will try to repro

s/input/output/

:)

i can dump real logs to an ip if you want

i _should_ just be able to give you yealink-raw.log lol

I'll let you know if it comes to that

I try to avoid it coming to that, though

I've gotten pretty good at debugging things via sample data, source code and configs

feel free to clone that repo

I may just

but for now, gotta go turn up traffic to a new dc!

(dayjob stuff; ls is just a hobby of sorts for me)

all im doing from ansible is copying /roles/aggregator/templates/logstash/* to /etc/logstash/conf.d

ok cool

nice one, thanks for your help

HI guys ... a stupid question .... should I use %{SPACE} for spaces in the logs or just I put a space ?