#haraka

did you add the plugin to the plugins file ?

no I wasn't sure where to add this / uncomment it?

do I also need to copy the relay.js file from the repo and put this in the config directory?

No - the .js file is the plugin.

You just need to add 'relay' to config/plugins to enable the plugin, then restart Haraka

does that plugin exist by default?

yes

Any plugin referenced in the base documentation is included with Haraka.

also, the smtp_forward plugin can validate recipient domains, by setting the check_recipient=true option in smtp_forward.ini

You just have to enable them.

then it will validate domains that are explicitly declared in smtp_forward.ini

ah ok, I thought I was missing the plugin because I couldn't see it using "ls" in the config directory

_smf_: I'm thinking of writing a plugin that adds the source IP of the sender to redis if rspamd score is bigger than <configurable score>

the goal being to pull that list into our own dnsbl

do you know if such a plugin already exists ? It's not very complicated but could be useful

last1: I'd just use the rspamd plugin ip_score instead.

last1, so can you end up with Gmail / Yahoo IPs in your DNSBL list?

when they let spam slip through?

well, I was thinking of setting a limit or something. if the IP is listed more than 100 times

like sometimes we get flooded for like 5-10 minutes with 1000 connections from the same IP

doing dictionary attacks/deliveries

Yeah - for once I agree with msimerson about this. You'll be setting yourself up for trouble unless you're really careful.

_smf_: does it matter where I add "relay" in the config/plugins file, I seem to remember order matters?

moatak787: it varies by plugin, but putting relay at the top should be fine.

last1, I use the rate limit plugin to greatly reduce the connection rates by bad/unknown remotes

Yeah - rate limiting is far better than trying to create a dnsbl.

throttle them to say, 10 connections / 5m, and you've mostly solved the same problem. After 5 minutes, their IPs are on the global blacklists anyway

not a bad idea. Thanks!

we already run our own dnsbl and thought it might be easier but throttling is good too

You'll need to fiddle with the limits quite a bit initially. Maybe have a 'unknown hosts' limit and a 'known hosts' limit which is higher.

agreed.

The other thing that you might be able to do which the stock plugin doesn't do is to monitor valid .vs. invalid recipients if you are able to and implement rate limiting based on those metrics.

I found the default limits not so helpful for my traffic until I extended them to support good/bad/unknown based on karma scores.

last1: as you aren't using greylisting - you'll have to come up with an alternative method for determining known .vs unknown hosts (e.g. hosts that actually have history sending mail to you).

what is the best way to debug why haraka is erroring when starting up, are there logs or logging available?

I'm getting the error: Failed to setup listeners: listen EACCES ::0:587

are you running as root ?

Lack of root permissions.

you need to be root to listen on port 587.

Or there is something already running on 587.

oh right.

ahh [facepalm]... forgot "sudo"

yeah it would be EPERM I think for the other one.

thanks

moatak787: make sure you set a user/group in smtp.ini to drop down to. You don't want to keep root privileges the entire time.

baudehlo: thanks for the tip, will check that we have this set

last1: you could for example store every IP you've seen each day in a Redis HyperLogLog for the last 40 days and create a combined view of these each day, dropping out the oldest one. Then you'd know for any given connection if you've spoken to that host in the last N days. You could also automatically remove IPs from the HLL unless they send you messages that you decide are not-spam.

Hmmm - I might try that myself.

lol

now it's getting more like what I was thinking

so if I have 100 connections from IP that all scored more than 25 on rspamd

and increment the redis cache with the total number of connections in the last 24h

and only list those IPs in the dnsbl

this way it's self cleaning

That would be more difficult. HLL's are cheap, storing every IP you see for 24 hours might not be so cheap.

for my traffic, storing every IP for 24 hrs is still cheap. very cheap

Anyway - unless you're having issues with load or having spam slip though, then I'd simply use the rspamd ip_score plugin and ignore it.

As they'll get auto-nuked by rspamd if they consistently send high scoring messages.

yeah, but I don't use rspamd for rejecting anything

I just score and pass on to other systems

LOL

There's one of your problems right there then ;-)

we're a really small provider...but have corporate customers. rejected e-mails are always bad

they have to end up in the Spam folder

I still see quite a few PFs from rspamd, but I haven't taken the time to customize / train it.

Right - I've got lots of corporate customers and I nuke stuff out at the SMTP level all the time.

It's how you go about doing it that counts.

assume the remote end can't interpret the bounce they receive

That's rubbish.

in our case, the Government of Canada

Even the dumbest user can tell a message wasn't delivered if they receive a bounce.

we have this customer, attorneys helping customers immigrate to Canada

expecting an automatic reply from the Government of Canada

...which got rejected..sender was an unmonitored mailbox

some rich family from somewhere got denied Visa to Canada

pretty messed up

Besides - stuffing 1,000 messages in a spam folder with a few false-positives in it is *way* worse, then a couple of bounce FPs because your user won't be able to spot the FPs mixed in with all the other junk without spending so much time searching, that you've instantly killed the whole reason for spam filtering in the first place.

yeah, that's another drawback

And quite honestly, that example you just gave is laughable. The attorney is at fault for sending a message with an un-monitored return-path for potentially important mail.

sort of, the e-mail was coming from the government

they were the ones misconfigured

and it took them something like 6-8 months to realize this happened. they asked for delivery logs and what not from 8 months previously

last1: so why was that individual message marked as spam and not all the other messages?

Or rather rejected.

I think it was a misconfigured server or something

if I recall correctly, government there has complicated names like: user@immigration.fs.on.gc.ca

and the server had its name to end with .local

btw, not sure if you saw in the rspamd channel

I modified your RCVD_TLS_LAST symbol

and added a MISSING_RCVD_TLS_LAST to add a score

LOL

yeah, we're scoring all those that don't deliver over TLS

been working like a dream :)

I wasn't sure whether to submit a pull request for that

No that isn't something I'd dream of doing myself.

But if it works for you - then cool.

if you have a chance, try and audit how many legit e-mails you still receive without tls

you'd be surprised

It wouldn't hurt to have that in the rspamd ruleset with a 0 score, so someone could enable it if they wanted to. And we can use it for corpus testing.

it was nothing major, just this: https://pastebin.com/pS3TqeXq

was it you that added the HTML_META_REFRESH_URL regexp to rspamd ?

last1: yeah

sorry, I don't have a forked version of rspamd

but you use this regex

re = '/<meta\\s+http-equiv="refresh"\\s+content="\\d+;url=/{sa_raw_body}i'

can you modify to: re = '/<meta\\s+http-equiv="refresh"\\s+content="\\d+;\s?url=/{sa_raw_body}i'

there might or might not be an empty space after content="0; url

Oh - right, sure thing.

why are double \\ used btw ?

last1: \ is used for escaping control characters in lua so \ requires escaping for literal \ - unless you use square quotes which aren't escaped: https://dpaste.de/pAGm/raw

so if you want to write \s you write \\s

to match empty spaces

for regex it's probably most comfortable to just use the square quotes; '\\s' == [[\s]]

alright, thanks. I'll continue this in the rspamd channel

shouldn't this lua regex: <meta\\s+http-equiv="refresh"\\s+content="\\d+;\\s*url=

hang on. does \s match tabs as well ?

it just says whitespace character

last1: yes.

[13Haraka] 15msimerson closed pull request #1868: Get pool (06master...06get_pool) 02https://github.com/haraka/Haraka/pull/1868

[13Haraka] 15msimerson closed pull request #1851: tls consistency cleanups (06master...061691-dry-tls.ini) 02https://github.com/haraka/Haraka/pull/1851

[13Haraka] 15msimerson pushed 1 new commit to 06master: 02https://github.com/haraka/Haraka/commit/c6381...

13Haraka/06master 14c638176 15Matt Simerson: tls consistency cleanups (#1851)...

[13Haraka] 15msimerson opened pull request #1869: merge master into v3 (06v3...06master) 02https://github.com/haraka/Haraka/pull/1869

[13Haraka] 15msimerson pushed 1 new commit to 06v3: 02https://github.com/haraka/Haraka/commit/38184...

13Haraka/06v3 143818462 15Matt Simerson: merge master into v3 (#1869)...

[13Haraka] 15msimerson closed pull request #1869: merge master into v3 (06v3...06master) 02https://github.com/haraka/Haraka/pull/1869

[13Haraka] 15msimerson opened pull request #1870: merge master into v2 (06v2...06master) 02https://github.com/haraka/Haraka/pull/1870

[13Haraka] 15msimerson pushed 1 new commit to 06v2: 02https://github.com/haraka/Haraka/commit/1d7bf...