SPAM Getting Through

Printable View

Show 40 post(s) from this thread on one page

10-21-2014, 11:55 AM
AdamB

SPAM Getting Through

Starting my own thread on this as we aren't getting the results we want. We have been having spam issues for months now. At first we thought it was due to our DNSBL issues, but we have had those fixed for quite some time and reset our database a number of times with no decrease in spam.

We also upgraded to untangle 11 with no decrease in spam. I even went as far as doing a complete re install to see if that would help and it hasn't.

We have opened a support ticket and have done everything suggested by support but honestly they aren't much help other than resetting the database (Ticket #30669), also note we haven't even got a response on this ticket since the 10th of this month. We have provided example after example with no real resolution so far. I understand that we will never get 100% SPAM blocking, but it just seems we should be able to prevent more than we are now.

Another change we did this week was set spam blocker to quarantine instead of mark. This way we could enable spamassassan on our mail server in hopes of training it with the "spam" and "not spam" buttons. I am unsure how far this will really get us, but it was worth a shot.

I am open to suggestions on just about anything to get this resolved. Our users are in a uproar over the issue and number of them are upper management to make matters worse. I just don't know what to do at this point other than look for a new solution which I do not want to do.
10-21-2014, 03:05 PM
AdamB

UPDATE: Untangle support contacted us back shortly after this was posted. We were told the following.

"It's just the way it is"

In my opinion this is not acceptable.

Then the support person told us we should disable tarpitting as it disables the bayes scoring. We never ran tarpitting till support advised us to enable it recently. As far as I can tell tarpitting is nothing more than some RBL's. Why in the world would RBL's disable the bayes scoring? Can anyone confirm or deny this?
10-21-2014, 03:07 PM
dmorris

http://wiki.untangle.com/index.php/S...t_is_tarpit.3F
10-21-2014, 03:09 PM
Danp

I feel your pain. That's essentially the same response we were given recently.
10-21-2014, 03:46 PM
sky-knight

Quote:

Originally Posted by AdamB

UPDATE: Untangle support contacted us back shortly after this was posted. We were told the following.

"It's just the way it is"

In my opinion this is not acceptable.

Then the support person told us we should disable tarpitting as it disables the bayes scoring. We never ran tarpitting till support advised us to enable it recently. As far as I can tell tarpitting is nothing more than some RBL's. Why in the world would RBL's disable the bayes scoring? Can anyone confirm or deny this?

Tarpitted e-mails don't get processed, if they aren't processed by the module the results aren't used in the bayes training. The same thing is true if you bypass egress smtp, those messages can't be seen to be learned from.
10-21-2014, 04:44 PM
AdamB

Quote:

Originally Posted by dmorris

http://wiki.untangle.com/index.php/S...t_is_tarpit.3F

Well that makes sense. Just wish we weren't told to turn it on in the first place. So is it a waste of our time to continue providing examples or is there actually a possibility we can make an improvement?
10-21-2014, 07:33 PM
dmorris

Quote:

Originally Posted by AdamB

Well that makes sense. Just wish we weren't told to turn it on in the first place. So is it a waste of our time to continue providing examples or is there actually a possibility we can make an improvement?

Submitting samples doesn't really help make an improvement - It helps us debug whats wrong.
Looking at the old samples you submitted - the bayesian gave some stuff a 0% chance of being spam, that commtouch indicated was spam.
That is suspicious and to me suggests that your bayesian has mistrained itself (usually bad DNS). They agreed and they reset the bayes. They reset it on Oct 3rd and asked you to monitor - if you have new samples you need to submit them. There aren't any samples in the case since September, and it looks like you've had your bayes reset, upgraded to 11.0 and reinstalled. The old ones are not relevent.

Also, a quick tests shows your DNS is flaky at best.
Spam Blocker is very DNS dependent. If DNS isn't working, it can't communicate with the cloud services.
I totally suspect your DNS is sometimes down, as some of the older samples show some pretty spammy stuff which matches not a single network-based test. For example, your Amazon spam example hits nothing, but for me it hits URIBL, RAZOR, and BRBL, and SURBL.
Its possible that those were added since, but I suspect it was just that DNS failed at the time of the original scan.

Quote:

[root @ untangle] ~ # for i in `seq 50` ; do echo "test $i" ; curl -s http://www.untangle.com/download/pat...spam_health.sh | bash | grep FAILED ; done
test 1
test 2
test 3
test 4
test 5
test 6
test 7
[207.14.235.234] : URIBL FAILED
test 8
test 9
[207.14.235.234] : URIBL2 FAILED
[207.14.235.234] : URIBL2 FAILED
test 10
test 11
[207.14.235.234] : URIBL FAILED
test 12
test 13
[207.14.235.234] : URIBL FAILED
[207.14.235.234] : SURBL FAILED
[207.14.235.234] : SURBL FAILED
[207.14.235.234] : SURBL FAILED
test 14
test 15
test 16
test 17
test 18
test 19
[207.14.235.234] : URIBL2 FAILED
[207.14.235.234] : URIBL2 FAILED
test 20
[207.14.235.234] : URIBL FAILED
test 21
[207.14.235.234] : SURBL FAILED
test 22
[207.14.235.234] : SURBL FAILED
[207.14.235.234] : SURBL FAILED
test 23
[207.14.235.234] : URIBL2 FAILED
[207.14.235.234] : URIBL2 FAILED
[207.14.235.234] : URIBL2 FAILED
test 24
test 25
[207.14.235.234] : URIBL2 FAILED
test 26
test 27
[207.14.235.234] : URIBL FAILED
[207.14.235.234] : URIBL FAILED
[207.14.235.234] : URIBL FAILED
test 28
test 29
[207.14.235.234] : URIBL FAILED
[207.14.235.234] : URIBL FAILED
test 30
[207.14.235.234] : URIBL2 FAILED
test 31
test 32
[207.14.235.234] : URIBL2 FAILED
test 33
test 34
test 35
test 36
[207.14.235.234] : URIBL FAILED
test 37
test 38
test 39
[207.14.235.234] : URIBL2 FAILED
[207.14.235.234] : URIBL2 FAILED
test 40
test 41
[207.14.235.234] : URIBL FAILED
test 42
test 43
test 44
[207.14.235.234] : URIBL FAILED
[207.14.235.234] : URIBL FAILED
test 45
test 46
test 47
test 48
test 49
10-21-2014, 08:03 PM
dmorris

I'm not sure who "207.14.235.234" is, but its pretty much completely borked.

It know I'm a broken record, but its not really worth investigating spam stuff until you get your DNS situation completely sorted. Untangle needs a completely functional and reliable internet connection.

Quote:

[root @ untangle] ~ # for i in `seq 50` ; do echo "test $i" ; curl -s http://www.untangle.com/download/pat...spam_health.sh | bash | grep FAILED ; done
test 1
[207.14.235.234] : DNS FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : SpamHaus FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
test 2
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
test 3
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
test 4
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
test 5
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
test 6
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
test 7
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
test 8
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamHaus FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)
[207.14.235.234] : SpamCop FAILED (exit code: 1)
[207.14.235.234] : SpamHaus FAILED (exit code: 1)
[207.14.235.234] : SpamHaus FAILED (exit code: 1)
[207.14.235.234] : URIBL FAILED (exit code: 1)

even just simple tests by hand show bizarre results:

Quote:

[root @ untangle] ~ # host 2.0.0.127.bl.spamcop.net 207.14.235.234
Using domain server:
Name: 207.14.235.234
Address: 207.14.235.234#53
Aliases:

2.0.0.127.bl.spamcop.net has address 127.0.0.2
[root @ untangle] ~ # host 2.0.0.127.bl.spamcop.net 207.14.235.234
Using domain server:
Name: 207.14.235.234
Address: 207.14.235.234#53
Aliases:

2.0.0.127.bl.spamcop.net has address 127.0.0.2
Host 2.0.0.127.bl.spamcop.net not found: 3(NXDOMAIN)
Host 2.0.0.127.bl.spamcop.net not found: 3(NXDOMAIN)
[root @ untangle] ~ # host 2.0.0.127.bl.spamcop.net 207.14.235.234
Using domain server:
Name: 207.14.235.234
Address: 207.14.235.234#53
Aliases:

2.0.0.127.bl.spamcop.net has address 127.0.0.2
[root @ untangle] ~ # host 2.0.0.127.bl.spamcop.net 207.14.235.234
Using domain server:
Name: 207.14.235.234
Address: 207.14.235.234#53
Aliases:

2.0.0.127.bl.spamcop.net has address 127.0.0.2
Host 2.0.0.127.bl.spamcop.net not found: 3(NXDOMAIN)
[root @ untangle] ~ # host 2.0.0.127.bl.spamcop.net 207.14.235.234
Using domain server:
Name: 207.14.235.234
Address: 207.14.235.234#53
Aliases:

2.0.0.127.bl.spamcop.net has address 127.0.0.2
Host 2.0.0.127.bl.spamcop.net not found: 3(NXDOMAIN)
10-21-2014, 10:52 PM
sky-knight

Code:

for i in `seq 50` ; do echo "test $i" ; curl -s http://www.untangle.com/download/patches/generic/check_spam_health.sh | bash | grep FAILED ; done

Dirk, I hope you don't mind but I'm snagging this for my tool box!

BTW 207.14.235.234 looks like a Century Link primary DNS server. I would assume that like Cox, it's over utilized.
10-22-2014, 03:11 AM
AdamB

Quote:

Originally Posted by dmorris

Submitting samples doesn't really help make an improvement - It helps us debug whats wrong.
Looking at the old samples you submitted - the bayesian gave some stuff a 0% chance of being spam, that commtouch indicated was spam.
That is suspicious and to me suggests that your bayesian has mistrained itself (usually bad DNS). They agreed and they reset the bayes. They reset it on Oct 3rd and asked you to monitor - if you have new samples you need to submit them. There aren't any samples in the case since September, and it looks like you've had your bayes reset, upgraded to 11.0 and reinstalled. The old ones are not relevent.

Also, a quick tests shows your DNS is flaky at best.
Spam Blocker is very DNS dependent. If DNS isn't working, it can't communicate with the cloud services.
I totally suspect your DNS is sometimes down, as some of the older samples show some pretty spammy stuff which matches not a single network-based test. For example, your Amazon spam example hits nothing, but for me it hits URIBL, RAZOR, and BRBL, and SURBL.
Its possible that those were added since, but I suspect it was just that DNS failed at the time of the original scan.

Fair enough I will have them submit more examples.

Show 40 post(s) from this thread on one page