#logstash

dino82: if the world goes to hell, it might be running a grok you weren't expecting. Set a unique tag_on_failure for each grok just to make sure.

Alright

What if I am still getting _grokparserfailures on logs that seem to grok perfectly?

could be a bug somewhere (logstash or the debugger), i suppose. let's confirm the first part first.

hello -- does anyone have any experience with the jdbc plugin?

I am trying to connect logstash to a sqlserver express instance on my local machine, but I keep getting an error: :message=>"Pipeline aborted due to error", :exception=>#<LogStash::ConfigurationError: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?>

Looking around online, it seems like that error could be thrown if any of the required jdbc configuration settings are off

is there a favorite site for pasting code here?

i see github gists and pastebin

<http://tinyurl.com/jtdqmak>; (at gist.github.com)

stuff that's commented out has been tried, in all sorts of different combinations

how can logstash handle its own logs? log to itself?

Title: TheP(aste)B.in - For all your pasting needs! (at thepb.in)

troll!

kick

Fuck mossad, Cia' Nsa, fake jews from occupy " israel

nettly: if you can, setup a separate LS/ES to handle logs from your main cluster.

rastro: backup LogStash/ElasticSearch ?

nettly: not a backup per-se, just another cluster to manage the ELK logs themselves. it can be a small cluster (mine have always been a single machine to run LS/ES).

rastro: so e.g. a 2nd cloud instance (pardon my ignorance) just for logging "horror" errors

where the whole cluster breaks down

nettly: there are less-than-horror errors, or hey-horror-might-coming errors.

might-be-coming :/

rastro: hm, thanks :)

rastro: out of your experience: with docker, would you use filebeat inside to get the logs to logstash? or directly from docker own logging?

rastro: and are you using cloud instances like on amazon for these clusters?

nettly: i'm happily docker-free. i've built two clusters with a management cluster on the side, and they've both been bare metal, but it would work in AWS, too.

rastro: so rather containers/docker you prefer/are using currently cloud/VM/baremetal

hm, in the end it rather counts how well the receipts were written :D - then it doesn't really matter if you put it into a container, onto a VM or right on baremetal, I guess

nettly: had one division using vm/baremetal and another on AWS.

rastro: your own baremetal servers? are you using something like kubernetes or openstacks for the hvms?

nettly: i know they were our own servers, but some other group set them up, so i don't know what vm software they were using. I got a shell prompt, which is all i cared about.

so kind of Virtual Server then

hm

rastro: hm, another thing: Would you try to get all applications/services log directly to logstash or at least to a log daemon (like syslog) - or use filebeat on log files?

rastro: using filebeat I see one danger that at some point the log file data occupies the server and it could even shut down

having all the log data directly passed to logstash would mean I only have one central place where log data accumulates and I have to wrory about disk space there.

nettly: you have to plan for when logstash (and/or ES) is down. It'll be down at least when you push a new config once in a while, and might be down longer due to some tech issue.

rastro: I see - so the logstash daemon may be able to hold data for some while?

nettly: so... filebeat is nice since it will remember where it left off on a file, whereas syslog will (IIRC) just drop the data.

I see

rastro: so I would set up some destructive log rotation for the log files so the very old log files can be purged

because that log data is quite probably already in ES

nettly: LS only accepts a limited amount of data into its buffers (20 records, IIRC).

oh

this can be full very quickly then, especially with a http server (nginx)

nettly: when it's full, FB won't be able to send it more data, so it will queue up on the FB side and all/most should be good

cool

rastro: but an auto-purge of very old log files or when a quote for these log files has been reached for that specific daemon would make sense?

otherwise the server may become full and stop functioning (it happened once - ok, I put already lots of stuff onto it - but still)

nettly: for our retention policy, we use daily indexes and delete them (with curator) after X days.

rastro: ah, so the log files that are used by filebeats are purged? or also the logs (that passed the curation) on ES?

nettly: system logs are usually rotated/deleted by logrotate. If you have other (app) logs, they would have their own rotation on the client side.

rastro: I see, so there are log files are usual with a rotation that purges the oldest log files - and the logs are stored on ES and curated for removal or keeping

nettly: sounds right.

rastro: thanks - this part always bothered me (growing log files somewhere outside es)

hi folks, logstash isn't keeping my beats port open very long, nothing in logs, no idea how to further troubleshoot, any direction?

rastro: Hm, one "special/exotic" attack comes to mind with logrotation: Imagine an attacker dos'es a server so it produces lots of log entries - those would go to file and then to logstash - and let's say the logstash/es breaks down, there may be the hypothetical point that the log files accumulate so fast that old log files are purged without being sent to ES - and then indeed some log data may be lost that could have been interest

nettly: yes, if your client-side log rotation happens when LS/ES are not accepting logs, you can lose stuff.

rastro: would it ever be possible that logs come so fast that filebeat cannot send them all fast enough to ES/ES process them that logrotation eats them? Or is this very unlikely because the bottleneck would be elsewhere?

nettly: sure, it's possible. but ES scales well (horizontally) to help with such problems.

rastro: ok, thanks :)

so, how to troubleshoot beats port closing a short while after logstash is started?

short while could be 10-30 minutes

ardya: i didn't think FB ports were permanent. you're implying that they don't reopen when this happens?

evidently not since a client beats is consistently sending, but the newer data is not received by logstash, nor does lsof show the port open

ardya: does FB log anything itself?

the client? no

ardya: seems like it should. there's also debug/verbose mode in FB that might give you more info.

hello rastro

is there a debug mode in logstash?

the problem isnt the client, its logstash

ardya: yes --verbose on the command line

unless they changed it

ardya: but what the client sees could help you debug it.

when i restart logstash, then the older data is reecived from the client