#logstash

to be honest, I don't know exactly :D I am searching for some kind of best practise to get my logs from servers into kibana to get them sorted by facilities, severities and so on

thats it

I think now I am stuck with syslog-ng again since it duplicates timestamp and hostname again

I got rid of this before, but now it came back whyever

last state was that I had it almost done but syslog-ng sent single lines of logs for 1 message

with that [meta sequenceId="4"] in it

single lines or multiple lines?

it were single lines with [meta sequenceId="4"] 5 6 7 in it .. belonging to the same log message

on the client(server) its one message

Is there a good input to check an http endpoint periodicly and send the info somewhere?

oli_144: and that's something you want to disable right?

in kibana I see 10 entries for 1 logentry with 10 lines

I was thinking something like websocket but that does not seem to have a periodic poll option.

doug_f: nothing built in for now, either have the webapp do the periodic calling through websockets, or maybe have a script that writes to a file the results every X minutes

jsvd : yes

is there a way to tell logstash to drop messages if queue is hits the limit?

oli_144: there seems to be a no-multi-line flag? http://www.balabit.com/sites/default/files/docu...

<http://tinyurl.com/pawds55>; (at www.balabit.com)

oli_144: I'm not that familiar with these things, is there a way in your app to log everything in just 1 line?

I am just trying to get "all" logs to kibana, normal linux syslogs

oli_144 from the message jsvd just sent you you should be able to change the way the syslog-ng operates so it doesn't split the messages in to multiple lines, so you don't have to fix them once recieved, I would agree with him that would be the easiest and best way to solve your issue...

I tried not to use the legacy network driver, but the newer syslog driver

since I already had the grok thing running for the newer RFC5424 syslogging

I have logs coming through at 16:25, the date stamps on the log are 16:25, but in logstash @timestamp is set to 15:25.... and I can't figure out why it is going to UTC time instead of UTC +1

date is correct (i.e. UTC +1) on both sending and logstash server

it is always UTC in elasticsearch

use the date filter to match them

Rumbles

I do use the date filter, but the date I want to use is taken from a value in the json of my messages, if the json parser doesn't work the messages go on to default

is there any way to change that/

infact

even on message that have gone through the date filter ES is still showing the date as UTC time, not the time set in the field that was parsed by the date filter

http://pastebin.com/YuvFR5kv so for example, here is my config, date filter matches off the timeStamp field, "timeStamp": "2015-04-01 16:35:01.665" and yet "@timestamp": "2015-04-01T15:35:01.665Z"

Title: timestamp is reverting to UTC not the time set by date filte - Pastebin.com (at pastebin.com)

so it has read the info from timeStamp and set it corectly (the ms are correct) yet the time has gone to UTC time instead of UTC +1 which is correct

ah, yeah i don't know if that's possible

you could make timeStamp a date field and sort on that instead of @timestamp i suppose

but each different log type takes it's timestamp value fro a different field

using @timestamp makes sense as that is common for all log types

however, changing the time to UTC doesn't make sense

kibana auto-corrects it to my local timezone so i don't mind

because for half the year the messages will be an hour out

how is that configured?

somewhere in the gui :)

is on by default i think, at least in kibana3 haven't looked at it in k4

maybe in the settings for the table?

I'm using k3