#logstash

when I entered the date match the config failed

paste the config somewhere if you can

I have a pastebin with the config in if you could spare a minute?

Title: Having trouble doing 2 things: 1) Get the timestamp from the CSV to be @timesta - Pastebin.com (at pastebin.com)

I'm updating with the date match now

the if statement seems wrong

just put if [timestamp]

oh

didnt read through, i though that was condition to get the date

I had the date { match => [ "timestamp", "UNIX_MS" ] }

under the last mutate statement

that made the config fail

was there any other error

did you use the configtest

No. I'm new to logstash/elk, and got it up and running through a docker container I found

Speaking of timestamps. "YYYY-MM-dd HH:mm:ss.SSS", will that match a timestamp with more than 3 digits in the millisecond part?

Norrland: don't hijack my thread :P

^^

there can be only 1000 milliseconds in a second :) i dont know if it takes smaller fractions than milliseconds

yybel: true. Some applications have more fine-grain timestamps with HH:mm:ss.SSSSSS :)

Tyr-Heimdal: no date {} part in your config?

<http://tinyurl.com/mclv6ms>; (at www.elastic.co)

Title: input { lumberjack { port => 5043 ssl_certificate => "/etc/ssl/logs - Pastebin.com (at pastebin.com)

This is my current config, that broke

it has the date part

gives me _csvparsefailure

so it gives the error at runtime?

if I remove the date-part, it parses the logs but ofc gives me the epoch as a number field

yeah

try renaming the timestamp in your csv and date filter to something else like epoch_time

i dont know if its mad because the name is almost same as @timestamp

its weird that its csvparsefailure

Tyr-Heimdal: ah.

same csvparsefailure

on all events?

hey

its the conversion to integer maybe

maybe it must be string

it only leaves 2 events from thousands

if I remove the date part, I get thousands of events

oh, date filter doc says unix_ms parses int value

yeah, read somewhere that it should be int

try anyways

same result. gives 2 entries

is there anything in logstash logs

i guess I need to have something in the output-part for that..?

no theres something in /var/log/logstash.log and .err

and .stdout

INFO: [logstash-148599b191b4-8-12176] started

{:timestamp=>"2016-06-29T09:40:00.629000+0000", :message=>"Trouble parsing csv", :source=>"message", :raw=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>, :level=>:warn}

{:timestamp=>"2016-06-29T09:40:00.741000+0000", :message=>"Trouble parsing csv", :source=>"message", :raw=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>, :level=>:warn}

only stdout.log there

is your csv string in message field originally?

if I remove the date-part, then it populates the message field with the original lines, yes

try putting some trash filter before the csv filter

if ([message] =~ /^#/ or [message] == "") { drop{} }

Hello, can anyone in here help me with design advise, i have question regarding logstash/elasticsearch scale to accommodate large volume message , i am in process of building security event center that will get mostly syslog from firewalls ,switch, and winlogbeats what is most recommended for given scenario what is the message volume to consider using kind of queue management , or there is tweak on logstash or elasticsearch side

still weird that nothing goes through when date filter is there

are you checking the output from elasticsearch?

you mean where do I get my errors/conclusions from?

yeah

kibana

so, yeah

does elasticsearch have some errors about incoming events

for logstash debugging its good to use output stdout

then tail the stdout logfile

so, with that filter you gave me, it still processes everything like it should

if I re-add date after csv, it fails

I have to say it's a good feeling that it wasn't an easy fix ^^

:)

on the other hand...if it was a quick fix...this "#%£$€ would be working now :P

i tried your data and config and it works for me

maybe the input is somehow messed because i used stdin input

I'll open the file in an editor and double check if there are anything hidden in it

http://pastebin.com/u9P0xePt thats what i got out of it with the sample data you had in pastebin

Title: [JSON] { "message" => "1465992921014,,29862.2,74,131,170,62.827533,12.805496,0 - Pastebin.com (at pastebin.com)

that's just horribly annoying

nope, no whitespaces, no special chars or anything else

and that was just a duplicate of my config file?

changed input to stdin and output to stdout

otherwise the same

that shouldn't be significant, right..?

no unless lumberjack contains already some field information

but if you get all the same fields as in that paste except for the @timestamp being wrong then it shouldnt be the issue

i do

do the output stdout at least for better debugging

you get the info out of events that dont make it to elasticseach

I'll look into that

thanks!

how about the geo-issue? got any input there?

kibana map doesnt work?

right

im using geoip filter and it creates array named location with two number values for lat and lng

I get this output "location": {

"lon": 12.80557,

"lat": 62.827551

}

could it be that easy...that it's called lon instead of lng?

my data doesnt have those field names

just two values in an array for location

yours seems to be valid way too

maybe you need to also convert location field into "geo_point"

you have location.lat and location.lon as float but location is something default

meaning..?

or maybe im totally lost

i dont think logstash understands that geo_point type

so. if you check in kibana settings and indices the list of fields i guess the location field is not geo_point type

that's correct

so how do I make that happen

?

i suppose you have to edit elasticsearch index template for logstash

ok. I'll look into that

thanks for all your help! Allways nice to talk to people willing to help out :D

http://pastebin.com/gRMLZmqL theres one example how the template looks like with geopoint type

Title: [JSON] template - Pastebin.com (at pastebin.com)

you can ask more at #elasticsearch :)

awesome! :D