#logstash

What's the state on custom patterns with logstash 2.2 ?

My grok seems to fail because It cannot read custom patterns but then I found some issues in github aand people discussed that some plugin would need to be installed?

or is the documentation correct?

If I wanted to filter a username out of a cookie, where would I do it? As in Filebeat or Logstash or somewhere else?

hoover_damm: thanks for the hint with filebeats. just switched one host and for the moment it looks more constant.

dsa: yes I have kibana running

and from there I have verified the count

t4nk747: ok sorry - i understood it wrong. the from where you have the count the actual lines/files are sended ?

t4nk747: and be aware that under some conditions multiline lines are dropped and you find them inside of your logstash.* logs

dsa: I used "grep -c file" to count the line from log

t4nk747: how do you ship the logs to logstash ?

and I have enabled rubydebug and sent the output to a file

dsa: Using file input

then the macthed the count of rubydebug file and "grep -c file" of log

Will kibana let me search through the log files aswell as visualisng them?

t4nk747: and you have no errors in your logstash logs that he discards log data

square1: yeah it does

I searched the runbydebug output and found no errors

dsa: thanks. About linking log entries based on a unique string?

t4nk747: sorry no clue. t use a different log shipper way and for me only if the time is not synced, logstash discards messages because of format or the shipper does something wrong it works.

dsa: if you create the right dashboards with what you want to see and you filter the fields the way you want it - yes it does

^^ square1:

I actually want to see the log entries, not too fussed on visualisng the data just yet

is someone using log4j input?

square1: sorry mate - maybe you hit the #kibana channel on this - logstash is just putting the data into ES - nothing more :)

no problem, thanks!

warkolm: hei do You got some sort of a tutorial how create own plugin for kibana

dsa: where are you based?

hi, i'm having problems with a date filter. it works correctly when running logstash from the cmdline using stdin as input, but when i switch to a beats input it suddenly fails to parse the date with "Invalid format: \"2016/03/31 00:00:15\" is malformed at \"/03/31 00:00:15\""

date filter is basically just date { match => [ "mytimestamp", "yyyy/MM/dd HH:mm:ss" ] }

darn, filebeats is completly different than logstash-forwarder :(

square1: berlin

serdar: it is - i cannot use it. will try now the beaver -> redis -> logstash indexer approach

all my log tags go missing. just grabs all logs and dumps them - but without much infos on which logs etc

is there any user here using zenoss output with centos?

i'm having probs with bunny/jruby

does the elasticsearch update api can use wildcards in version 1.7 ?

so no one is using zenoss?

it seems that i might have been working before jruby1.9

Hello, I'm setting up an elk stack and am in the progress of doing some logstash config at the moment. I'm using logstash with the beats input to send all my server logs (/var/log/* + mysql + apache) to logstash with filebeat. At the moment I'm looking at logstash filters to parse date and other fields from syslog log files. Is it better to leave @timestamp and set another field with the syslog date or sh

ould I try to override @timestamp with the parsed date from syslog files?

donnex: i usually overwrite it

then @timestamp is the timestamp of the log and not the timestamp when the log was ingested

but fundamentally it's a matter of preference

Xylakant: thanks, I'll use that approach as well

looks like rubydebug { metadata => true } doesn't work here (i.e. the metadata =>true doesn't).

logstash 2.2.2

what's wrong?

Title: $ cat logstash-stdio.conf input { stdin {} } output { stdout { codec - Pastebin.com (at pastebin.com)

there. no metadata in the output...

hello