#logstash

BaM`: Thanks for the reply. I'm just trying to index log events but sucking at it right now. I have filebeat sending to logstash sending to ES. But the filebeat clients are constantly being stopped from logging with this error: 2018-02-26T18:19:26-08:00 ERR Connecting error publishing events (retrying): read tcp 10.0.10.19:51852->10.0.10.22:5044: i/o timeout

I'm just trying to figure out what I'm missing. For a while our ES was slammed. But now it's totally idle. And it's not getting any logs.

that sounds like anetwork issue between filebeat and logstash

*a network

Yeah, I thought so too. But nc -v connects successfully every time. I'm just about to give this thing a public IP (not behind the firewall) just to see if maybe we are running out of some resource on the firewall but I really don't think that's it. I just now realized that I'm running logstash with the default configs. I wonder if it needs more memory. Java really bugs me in that way. Things mysteriously perform like crap until you figure out the m

anyone have issues with policykit failing logstash?

Anyone happen to know if the IMAP input plugin (https://www.elastic.co/guide/en/logstash/curren...) allows a user to download an email's attachments?

maslen: I would assume so, as there's a strip_attachements option

but what you'd do with the attachemnt is questionable, unless it's plaintext I guess

BaM`: Download it.

I'm using it in prod, but the mailbox I'm polling never gets attachements, so I don't know for sure

Poop.

maslen: to where? what will logstash do with it?

I set up a docker-compose instance via https://github.com/deviantony/docker-elk and tried using IMAP access to a gmail account, but I don't see the emails even displaying

(First time ever using ELK, so I'm brand new to this all. I have used Splunk in the past, but never managed an isntance)

BAM`: The use case is that we'd use ES to store the full contents of all enterprise emails for 30 days or so.

The idea being that if an IDS alert fires, an analyst should be able to retrieve the full contents of the email that triggered the alert

ok so I've never tried this but I'll assume that you'd need to store them as text-encoded

also be careful if you're using a message broker - I know that kafka won't accept messaged >1m or so by default

*messages

elasticsearch will allow you to store binaries, but idk if logstash is capable of doing that - I'm going to assume not

Any easy way to test it? I have the gmail IMAP account set up

I just don't know how to debug the lack of content

are you sure you really want to put them in ES, on the offchance you might want to look at them later?

Any better ideas?

try input { imap { (...blah...) } output { stdout { codec => rubydebug } }

what's the mail service, for a start?

how about replicating the emails in question to another mailbox that users can't delete emails from?

input { imap { host => "imap.gmail.com" user => "test@mydomain.com"; password => "SECRET" check_interval => 30 } }

then just use a mail client to fetch the ones you want?

I created a new account to test it

the only reason I'm using the imap input is we have system-generated emails I want to parse info from

I guess the question is *why* do you want to put it into elasticsearch in the first place?

The attachments? Or the emails?

any of it

"The idea being that if an IDS alert fires, an analyst should be able to retrieve the full contents of the email that triggered the alert"

sure

I work in incident response

the enterprise gets >100k emails/day

but why not just put them into another mailbox on a mail server that's designed to handle emails and attachements?

unless you want to use elasticsearch to search for actual stuff in the email, there's not a lot of point putting the emails there in the first place

Oh, of course.

the IDS alerts fire on particular hostnames, senders, etc.

content within the emails or headers

right ok that's starting to make more sense

as I say I've never tried with attachements, but what I suspect is going to hapen is you'll have a big text-encoded field in your document

so you'll most likely have to extract that and decode it if you want to actually see the attachment

that's fine

Attachment data won't be needed all that often. And I can always write a Python script to decode it

so are you going from logstash directly to elasticsearch?

Uhh, I think so?

I'm using https://github.com/deviantony/docker-elk

Just trying a quick mock-up to see what it looks like when it ingests mail

right, yeah I guess you are

try using the stdout config I wrote back there

or maybe use the file output with the json codec

that will give you a good idea of what it's going to put in the ES index

Can I have multiple "outputs" ?

sure

but every message will hit every output unless you put a condition around them

and if one output blocks, they all will

unless you use multiple pipelines, but probably best to leave that for now

maslen: How would one use the imap input plugin? Never heard of it before. Been away from ELK for a good 6 years. A lot has changed! The description on the page for the plugin says "Periodically scan an IMAP folder (INBOX by default) and move any read messages to the trash." but I don't see where logstash or ES are involved there. That sounds like what imapfilter does.

Is ES being used as a mail store these days?

OK, I see in teh logs that it pulled the email contents

what are your outputs doing?

Meanwhile, I'm still trying to figure out why my logstash isn't accepting events.

tessier: are you using tls?

Poop. It can handle attachments - but only if they're plaintext.