Anybody insterested in my SPAM Blocker improvements?

[juank]

************************************************** *******
PLEASE DO NOT USE THESE SCRIPTS ANYMORE... I did this when 5.0.1 was out and not we are in 5.2. Some of the hacks are already included in 5.2. I'll be posting a new THREAD for 5.2. Please keep tuned! :juank:

************************************************** *******





If you want the new script for 5.2, please go to http://forums.untangle.com/showthread.php?t=3571


I modified our current/production untangle bridge trying to catch more virus/spam/phish messages.

Our new untangle server is also running DCC, rules from SANESecurity (for ClamaV) and rules from RulesEmporium (Rules du Jour for SpamAssassin) and after Untangle was letting pass 1,200 SPAM messages/week (even with EXTREME threshold in the SPAM Blocker) to the internal network, now is only passing about 100.

Before Untangle:

http://i76.photobucket.com/albums/j13/jcmachadouga/untangle/before_untangle.jpg

After Untangle - Defaults:

http://i76.photobucket.com/albums/j13/jcmachadouga/untangle/after-ungangle-defaults.jpg

MODIFIED Untangle:

http://i76.photobucket.com/albums/j13/jcmachadouga/untangle/modified-untangle.jpg

Is anybody out there interested in trying it? If so, please let me know.

Thanks.

[gotkimchi]

juank, I think most people would like to have it, however, they want a simple one click installation process.

[Antimidas]

I am extremely interested in an updated spam filter.

After one week of running for the new pro customer, 99.11% of all incoming mail is flagged as spam. This should be accurate as the domain was purchased from a prior owner and the new email addresses have not been published to outside parties.

However, even with the high rate of spam being caught, I noticed several looped bounce messages in the Exchange 2007 queues that I manually deleted. They were directly related to undeliverable spam messages that made it past the firewall.

Hopefully, the Untangle will keep spam manageable. There is not a version of MailEssentials and MailSecurity from GFI for Exchange 2007 at present. Rather, it required an Exchange 2003 front-end server running SMTP and passing the messages back to the internal information store. That is an added cost that the customer cannot yet justify. For now, everything seems to be under control with just the Untangle.

An additional feature would be the ability to manually process the spam cache to permit blocked mail through if it is determined that it is not spam. I am not sure that it is a feature that is forthcoming or that I have overlooked.

[mdh]

juank,

Are the SANESecurity and RulesEmporium rules open-source or are they available at a cost? That may make a difference to people, as would the ease/difficulty of integration. Go into some more detail!!! It would obviously be of value to all whether or not they choose to go that route in the long run.

[juank]

OK, here they are.

First of all, I'm not a developer and I'm not inventing or creating anything, I'm just adding some extra features to the Spam Filter, antivirus/phish modules.

I was going to create an automatic script, but I prefer people knowing what are they doing. Run/do this at your own. I've been running my new modified system for a month now and it's working great, you will see improvements, I promise.

Just open a shell window and run one by one.


# First, stop the untangle VM, just in case.
/etc/init.d/untangle-vm stop

cd /tmp
wget http://oss.itos.uga.edu/uthack.tar.gz
gunzip uthack.tar.gz
tar -xvf uthack.tar
cd ut-hack


cp sanesecurity.sh /var/lib/clamav

mkdir /etc/rulesdujour
cp config /etc/rulesdujour
cp rules_du_jour.bash /usr/share/spamassassin
cp -v -R etc-spam/spamassassin/* /etc/spamassassin/

#Make sure rules_du_jour sends reports everytime it updates itself
# Please replace "your_email_here" with your current email address (no quotes needed)

cd /usr/share/spamassassin
sed 's/juan@itos.uga.edu/you_email_here/g' rules_du_jour.bash > rules_du_jour.bash.tmp
cp -u rules_du_jour.bash.tmp rules_du_jour.bash


echo "deb http://ftp.de.debian.org/debian/ sarge main contrib non-free" >>/etc/apt/sources.list
apt-get update
apt-get install build-essential
export PATH=$PATH:/usr/bin/gcc

cd /tmp
wget http://www.rhyolite.com/anti-spam/dcc/source/dcc.tar.Z
gunzip dcc.tar.Z
tar -xvf dcc.tar
cd dcc-*
./configure --disable-dccm
make install
cdcc "delete 127.0.0.1"
ln -s /var/dcc/libexec/cron-dccd /usr/bin/cron-dccd
cd /tmp/ut-hack

cp dcc/updatedcc /var/dcc/libexec
cp dcc/DCC /etc/init.d/
cp dcc/dcc_conf /var/dcc

update-rc.d DCC defaults


echo "5 * * * * root /var/lib/clamav/sanesecurity.sh >> /tmp/sanesecuity.log" >>/etc/crontab
echo "00 02 * * * root /usr/share/spamassassin/rules_du_jour.bash >> /tmp/rules_du_jour.log" >>/etc/crontab
echo "0 5 * * * root /usr/bin/cron-dccd" >>/etc/crontab

# restore your apt sources.list
cd /etc/apt
grep -v debian sources.list > sources.list.tmp
cp -u sources.list.tmp sources.list





# re-start your system

reboot

# enjoy.


Please send me your comments/improvements.

Thanks

[juank]

So .... nobody wants to test it? :(

[Silver Bullet]

Wow juank, do you work in sales?:D

[dmorris]

To give an update - we broke this down into several separate bugs currently to be put into 5.1

If all goes well in QA - juank's changes will be released in Thunderbird! :D

http://bugzilla.untangle.com/show_bug.cgi?id=3308
http://bugzilla.untangle.com/show_bug.cgi?id=3310
http://bugzilla.untangle.com/show_bug.cgi?id=3317

[juank]

Thanks a lot Dirk!

[jcyphert]

Just curious because I don’t know shell scripts that well, If we mess something up can the restore from backup correct it?

[mdh]

The backup and restore are Untangle configurations, not full system. You mess with the shell and you may have problems unless you have documented or made backup copies of that which you have messed with.

[juank]

It's not going to mess up your system. It will only add some files/rules to catch more spam/virus.

[mdh]

Not what I meant. If you did a Untangle reinstall, your mods would be gone and they would have to be reinstalled.

[juank]

MDH is right.

For example when I did update my untangle to 5.0.3.1-1 I lost my "changes". I had to run the script again.

Thanks.

[mdh]

Sorry for the inference. (That's my big word for Thursday!)

[juank]

No reasons to be sorry. I appreciate your input. ;)

[chounmin]

I have problem with the above script. The DCC is not created and the following has some error.

"echo "deb http://ftp.de.debian.org/debian/ sarge main contrib non-free" >>/etc/apt/sources.listls 50"

Cheers

[juank]

Sorry. I just found a bug in the script... I meant, a "keyboard bug" ...

I replaced/fixed the line and put it in the website again.

Please go to the first page of this posting and copy the code again. I just updated it. Thanks



Please contact me if you have any question.

[dmorris]

Actually, could you edit the script to return the sources.list to the original after it runs.

Otherwise auto upgrade will get stuff from debian and may mess up your server.

[wickster215]

Actually, could you edit the script to return the sources.list to the original after it runs.

Otherwise auto upgrade will get stuff from debian and may mess up your server.

Any details on how to do this juank?
Also, is there anything that those of us who are not very good with linux may need to look for in the future before we put this into a production environment?

[juank]

Yes, I did I just updated my post with the code.

I basically added this:

# restore your apt sources.list
cd /etc/apt
grep -v debian sources.list > sources.list.tmp
cp -u sources.list.tmp sources.list

[wickster215]

Ok well, its only been 1 day, but so far with these changes... In my one test account at home, untangle with your changes caught 78 email and let only 1 through.
Not quite enough of a sample to put this into a production environment for me yet, but its looking good so far.

[dmorris]

keep us updated on your stats

great stuff with this script juank! :D

[juank]

Thanks.

I'm just trying to help other folks like me dealing with the same problem. what I feel frustrated (sometimes, not always) is trying to get some basic information from you guys ( I mean the developers ).

For example: Does the attack_blocker uses SPAMCOP and/or SURBLs?

Thanks

[chounmin]

Here are my situation.

Hardware:
- Intel(R) Pentium(R) D CPU 2.80GHz / 4Gb of ram

Services:
- Spamblocker
- Virus blocker
- Web filter
- Spyware
- Firewall
- Router
- Untangle Reports

Users:
- about 400

1. The changes cut down quite a bit of spam coming thru to my mail server.
2. But the memory ultilization kind of spike up > 3 G after running for 12 hours.
3. Once that happens, web filter is not working and mails starts to delay.

Hope someone can shred some light how to keep spam low and yet letting all service running well.

[juank]

chounmin,

Can I get a screen shot of your top command? I don't have any problem with memory utilization after the changes.

Your system should get all the updates automatically after a period of about 48 hours.

If you can, please send me a "ls -l" of your /etc/spamasssin and /usr/share/spamassasin directories via a PM.

Thanks

[juank]

I just updated the top of this post with some graphs to show the improvements.

Thanks

[chounmin]

How to configure to display those graph above?

[Silver Bullet]

I think juank uses "Big Momma" or "Big Sister" or "Big Daddy" or something.:D

[Pete]

Hi Juank,

I have entered your script letter by letter but when I get to the follow lines I get a file or "directory doesn't not exist"

cp dcc/updatedcc /var/dcc/libexec

cp dcc/DCC /etc/init.d/

cp dcc/dcc_conf /var/dcc



update-rc.d DCC defaults

And indeed there's no dcc directory in the /tmp folder and I'm not quite sure what do do next. I have repeated the commands from "wget http://www.rhyolite.com...." do see if I have missed something but I get the same result everytime.

Any clues?

Pete

[juank]

Hello Pete,

Are you sure you did this to start:

cd /tmp
wget http://oss.itos.uga.edu/uthack.tar.gz
gunzip uthack.tar.gz
tar -xvf uthack.tar
cd ut-hack

?

I tried it again and again, and I'm not having your problem. Please send me a PRIVATE MESSAGE if you want.

[juank]

I think juank uses "Big Momma" or "Big Sister" or "Big Daddy" or something.:D

I use Amavis.

[jcyphert]

juank,

I just inserted your script and now the untangle server can not be found... any ideas?

[Silver Bullet]

Wow. The server just disappeared? As in vanished? Have you looked under the desk? :D Sorry, couldn't resist.

[jcyphert]

:(
no, no... the update section that communicates with the untangle library server shows "no connection".

[jcyphert]

I must have changed the url of it some how in the update process. Everything seems to be working good aside from the library items not showing up

[juank]

The library items not showing up it's a bug in Untangle, I'm sure.

I have another test box, fresh installed and that happens sometimes. I just even reproduced the problem in 2 VMWare Untangle installs I have here...

[jcyphert]

is there a fix? it was ok before the changes. would a reboot take care of it?

[juank]

I'm telling you... it's kind of an Untangle bug.

Check one of my test boxes: It shows NO CONNECTION 2 out of 10 times I restart the GUI.

http://i76.photobucket.com/albums/j13/jcmachadouga/untangle/no-connection.jpg

[jcyphert]

yep, that's it.

Thanks juank!

[juank]

yep, that's it.

Thanks juank!

and that box doesn't have my modifications yet.

[Pete]

Yes I did the first part, and indeed there is a dcc directory in the ut-hack folder with the 3 missing files, updatedcc, DCC and dcc_conf. But they are in /tmp/ut-hack/dcc and not in /tmp. Is the location important? Can I do the cp operations with these files instead?

Thanks
Pete

[juank]

Yes I did the first part, and indeed there is a dcc directory in the ut-hack folder with the 3 missing files, updatedcc, DCC and dcc_conf. But they are in /tmp/ut-hack/dcc and not in /tmp. Is the location important? Can I do the cp operations with these files instead?

Thanks
Pete

Yes, and it's really important! Thanks! That's why I wanted people testing my script, so smart people like you find my typos...

I just updated the code on the first page of this posting.

Again, thanks a lot.

[Pete]

After doing the last changes and a reboot I now have more or less 0% of spam slipping through. Excellent work Juank!! Well worth the extra typing!! :)

Cheers
Pete

[juank]

Excellent! That's what I want to hear !
Please help me spread the word so we have more people testing it!

Thanks

[RichieIDI]

Juank,

I just installed Untangle for the first time on friday and just installed your updates two hours ago. Easy as pie (other than me spelling debian debain.).

I will have no way of telling if your improvments help because of the time frame, but I'm sure it will.

I do have a question, and maybe you can answer, maybe not. (I know you are just a user and all). Can you edit a spamassassin config to write into the headers why it scored as such? Or does untangled use spamassass just as a "Here's some text! What do you score it?" "Eh, 1.5" so it doesn't actually touch the emails and can't edit headers and etc.

[juank]

That's a good question I'm trying to find an answer too.

When I asked Untangle engineers, they told me they just use SA to give an score and then the Untangle VM (Java) decides what to do with it.

I really want to add score headers like I'm doing on my all system but ... I haven't find a way... not yet...

You mean something like this, right:

Microsoft Mail Internet Headers Version 2.0
Received: from xxxx ([128.192.1xxx]) by xxxx with Microsoft SMTPSVC(6.0.3790.3959);
Mon, 19 Nov 2007 14:00:28 -0500
Received: from xxxx ([128.192.xxxx]) by xxxx with Microsoft SMTPSVC(6.0.3790.3959);
Mon, 19 Nov 2007 14:00:27 -0500
Received: from localhost (localhost.localdomain [127.0.0.1])
by xxxxx (Postfix) with ESMTP id 26BD71D80E7
for <xxx@xxxx>; Mon, 19 Nov 2007 14:08:36 -0500 (EST)
Received: from cxxxx ([127.0.0.1])
by localhost (charon [127.0.0.1]) (amavisd-new, port 10024) with LMTP
id 29458-03 for <xxx@xxxx>; Mon, 19 Nov 2007 14:08:19 -0500 (EST)
Received: from fmailhost02.isp.att.net (fmailhost02.isp.att.net [204.127.217.102])
by xxxx (Postfix) with ESMTP id 0AA491D80E8
for <jxxx@xxx>; Mon, 19 Nov 2007 14:08:13 -0500 (EST)
Received: from talleytoropc (adsl-158-124-117.mia.bellsouth.net[68.158.1xxx])
by bellsouth.net (frfwmhc02) with SMTP
id <20071119190001H020009mvae>; Mon, 19 Nov 2007 19:00:02 +0000
X-Originating-IP: [68.xxx.117]
From: "Talley xxx" <xxxx>
To: "'Juan Machado'" <xxx@xxx>
Cc: "'Londono, Clara'" <xxx@xxxx.us>
Subject: Neighborhoods--Talley
Date: Mon, 19 Nov 2007 13:59:40 -0500
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="----=_NextPart_000_0031_01C82AB4.6E4D78F0"
X-Mailer: Microsoft Office Outlook, Build 11.0.5510
X-MimeOLE: Produced By Microsoft MimeOLE V6.0.6000.16545
Thread-Index: Acgq3lZg+N7aRhKGTLe3bhOGAHp1Rw==
Message-Id: <20071119190814.0AA491D80E8@charon.itos.uga.edu>
X-Virus-Scanned: amavisd-new at itos.uga.edu
X-Spam-Status: No, hits=2.689 tagged_above=-999 required=6.31 tests=AWL,
BAYES_00, DNS_FROM_RFC_ABUSE, DNS_FROM_RFC_POST, FR_BR_AFTER_HTML,
HTML_MESSAGE, HTML_TEXT_AFTER_BODY, MSGID_FROM_MTA_ID, SPF_SOFTFAIL
X-Spam-Level: **
Return-Path: talleytoro@charter.net
X-OriginalArrivalTime: 19 Nov 2007 19:00:27.0951 (UTC) FILETIME=[72F033F0:01C82ADE]

------=_NextPart_000_0031_01C82AB4.6E4D78F0
Content-Type: multipart/alternative;
boundary="----=_NextPart_001_0032_01C82AB4.6E4D78F0"

------=_NextPart_001_0032_01C82AB4.6E4D78F0
Content-Type: text/plain;
charset="windows-1250"
Content-Transfer-Encoding: quoted-printable

------=_NextPart_001_0032_01C82AB4.6E4D78F0
Content-Type: text/html;
charset="windows-1250"
Content-Transfer-Encoding: quoted-printable


------=_NextPart_001_0032_01C82AB4.6E4D78F0--
------=_NextPart_000_0031_01C82AB4.6E4D78F0
Content-Type: image/jpeg;
name="image001.jpg"
Content-Transfer-Encoding: base64
Content-ID: <image001.jpg@01C82AB4.6D7B82F0>

------=_NextPart_000_0031_01C82AB4.6E4D78F0
Content-Type: image/jpeg;
name="image002.jpg"
Content-Transfer-Encoding: base64
Content-ID: <image002.jpg@01C82AB4.6D7B82F0>

------=_NextPart_000_0031_01C82AB4.6E4D78F0
Content-Type: image/jpeg;
name="image003.jpg"
Content-Transfer-Encoding: base64
Content-ID: <image003.jpg@01C82AB4.6D7B82F0>


------=_NextPart_000_0031_01C82AB4.6E4D78F0--

I'll keep you posted.

[RichieIDI]

X-Spam-Status: No, hits=2.689 tagged_above=-999 required=6.31 tests=AWL,
BAYES_00, DNS_FROM_RFC_ABUSE, DNS_FROM_RFC_POST, FR_BR_AFTER_HTML,
HTML_MESSAGE, HTML_TEXT_AFTER_BODY, MSGID_FROM_MTA_ID, SPF_SOFTFAIL
X-Spam-Level: **

Yup, exactly that. And dang, if you can't figured it out, I stand no chance :)

[juank]

X-Spam-Status: No, hits=2.689 tagged_above=-999 required=6.31 tests=AWL,
BAYES_00, DNS_FROM_RFC_ABUSE, DNS_FROM_RFC_POST, FR_BR_AFTER_HTML,
HTML_MESSAGE, HTML_TEXT_AFTER_BODY, MSGID_FROM_MTA_ID, SPF_SOFTFAIL
X-Spam-Level: **

Yup, exactly that. And dang, if you can't figured it out, I stand no chance :)

Don't say that, I'm not an expert!

[RichieIDI]

I've used linux for... oh.. let me count it... 10 days now.

Other than one quick class in college and some other experimental days in my youth :)

You know how long it took for me to figure out how to ssh into my untangled box?

[etrigan63]

I tried the script and all was cool until I hit ./configure. I got the following:

loading cache ./config.cache
Rhyolite Software DCC 1.3.73
checking for cc... cc
checking for gcc... (cached) cc
checking whether the C compiler (cc ) works... yes
checking whether the C compiler (cc ) is a cross-compiler... no
checking whether we are using GNU C... (cached) yes
checking whether cc accepts -g... (cached) yes
checking how to run the C preprocessor... (cached) cc -E
checking for pthread_mutex_lock... (cached) yes
checking for gcc __attribute__... (cached) yes
checking sizeof(long)... (cached) 4
checking for gmake... yes
checking for xargs... (cached) /usr/bin/xargs
checking for wget... (cached) /usr/bin/wget
checking for perl... (cached) /usr/bin/perl
checking for sendmail... (cached) /usr/sbin/sendmail
checking for htpasswd... (cached) /usr/bin/htpasswd
checking for rrdtool... (cached) /usr/local/bin/rrdtool
checking for sa_len... (cached) no
checking for socklen_t... (cached) yes
checking for AF_LOCAL... (cached) yes
checking for ANSI C header files... (cached) yes
checking for pid_t... (cached) yes
checking for u_int32_t... (cached) yes
checking for u_int64_t... (cached) yes
checking need for string.h... (cached) no
checking whether time.h and sys/time.h may both be included... (cached) yes
checking for strings.h... (cached) yes
checking for paths.h... (cached) yes
checking for sys/cdefs.h... (cached) yes
checking for pthread.h... (cached) yes
checking for arpa/nameser.h... (cached) no
checking for resolv.h... (cached) no
checking for utime.h... (cached) yes
checking for vsyslog... (cached) yes
checking for getipnodebyname... (cached) no
checking for getipnodebyaddr... (cached) no
checking for freehostent... (cached) no
checking for getaddrinfo... (cached) yes
checking for getnameinfo... (cached) yes
checking for freeaddrinfo... (cached) yes
checking for gai_strerror... (cached) yes
checking for hstrerror... (cached) yes
checking for _res... (cached) yes
checking for res_init... (cached) no
checking for res_query... (cached) yes
checking for dn_expand... (cached) yes
checking for inet_ntop... (cached) yes
checking for inet_aton... (cached) yes
checking for gethostid... (cached) yes
checking for localtime_r... (cached) yes
checking for gmtime_r... (cached) yes
checking for timegm... (cached) yes
checking for altzone... (cached) no
checking for futimes... (cached) yes
checking for setpgid... (cached) yes
checking for poll... (cached) yes
checking for strlcpy... (cached) no
checking for strlcat... (cached) no
checking for getprogname... (cached) no
checking for daemon... (cached) yes
checking for getifaddrs... (cached) yes
checking for freeifaddrs... (cached) yes
checking for md5.h... (cached) no
checking for sysconf(_SC_PHYS_PAGES)+sysconf(_SC_PAGESIZE)... (cached) yes
checking for __progname... (cached) yes
checking FD_SETSIZE to compute NUM_CWFS... (cached) 32
checking run directory... (cached) /var/run
checking for sysctl(kern.boottime)... (cached) no
checking for IPv6... yes
look for sendmail milter library in ./../sendmail/obj.Linux.2.6.16-ck11-untangle-cd-486.i686
*** cannot build dccm without sendmail headers in ./../sendmail
and libraries in ./../sendmail/obj.Linux.2.6.16-ck11-untangle-cd-486.i686 ***
checking for Rsendto... (cached) no
creating ./config.status
.................................................c reating include/dcc_config.h
include/dcc_config.h is unchanged


What did I do wrong?

[RichieIDI]

Not sure you did anything wrong, that looks like mine.

[juank]

There is nothing wrong, it's just a warning.

Dccm is the Sendmail interface, you don't need that since it's only
SpamAssassin that uses DCC, not Sendmail. You only need dccproc and
optionally dccifd if you want to run it as a daemon.

What I just did, I modified (and posted) the original script to add "--disable-dccm" to your configure line and it should pass.

./configure --disable-dccm

If you want, run it again.

Anyway, the script on the FIRST page of this posting is updated now.

Again, Thanks a lot for all you folks giving me ideas to improve the script.

Thanks!

[etrigan63]

No joy. Make install fails with an error.


/tmp/dcc-1.3.73 # make install [root @ wally]
cc -D_FILE_OFFSET_BITS=64 -I../include -c -o aop.o aop.c
In file included from /usr/include/errno.h:36,
from ../include/dcc_defs.h:55,
from ../include/dcc_clnt.h:45,
from aop.c:42:
/usr/include/bits/errno.h:25:26: linux/errno.h: No such file or directory
In file included from /usr/include/bits/posix1_lim.h:130,
from /usr/include/limits.h:144,
from /usr/lib/gcc-lib/i486-linux/3.3.5/include/limits.h:122,
from /usr/lib/gcc-lib/i486-linux/3.3.5/include/syslimits.h:7,
from /usr/lib/gcc-lib/i486-linux/3.3.5/include/limits.h:11,
from ../include/dcc_defs.h:71,
from ../include/dcc_clnt.h:45,
from aop.c:42:
/usr/include/bits/local_lim.h:36:26: linux/limits.h: No such file or directory
In file included from /usr/include/sys/socket.h:35,
from /usr/include/netinet/in.h:24,
from /usr/include/netdb.h:28,
from ../include/dcc_defs.h:72,
from ../include/dcc_clnt.h:45,
from aop.c:42:
/usr/include/bits/socket.h:305:24: asm/socket.h: No such file or directory
In file included from ../include/dcc_defs.h:74,
from ../include/dcc_clnt.h:45,
from aop.c:42:
/usr/include/sys/param.h:23:26: linux/limits.h: No such file or directory
/usr/include/sys/param.h:24:25: linux/param.h: No such file or directory
In file included from ../include/dcc_clnt.h:45,
from aop.c:42:
../include/dcc_defs.h:186: error: `PATH_MAX' undeclared here (not in a function)
In file included from aop.c:42:
../include/dcc_clnt.h:97: error: `MAXHOSTNAMELEN' undeclared here (not in a function)
../include/dcc_clnt.h:188: error: `MAXHOSTNAMELEN' undeclared here (not in a function)
../include/dcc_clnt.h:353: error: `MAXHOSTNAMELEN' undeclared here (not in a function)
make[1]: *** [aop.o] Error 1
make: *** [all] Error 2
/tmp/dcc-1.3.73 # cd .. [root @ wally]
/tmp # rm dcc-* [root @ wally]
rm: cannot remove `dcc-1.3.73/': Is a directory
/tmp # rd dcc-* [root @ wally]
zsh: command not found: rd
/tmp # rmdir dcc-* [root @ wally]
rmdir: `dcc-1.3.73/': Directory not empty
/tmp # cd dcc-* [root @ wally]
/tmp/dcc-1.3.73 # cd include [root @ wally]
/tmp/dcc-1.3.73/include # dir [root @ wally]
dcc_ck.h dcc_heap_debug.h dcc_proto.h kludge.h.in
dcc_clnt.h dcc_ids.h dcc_xhdr.h sendmail-sysexits.h
dcc_config.h dcc_ifaddrs.h dccif.h
dcc_config.h.in dcc_md5.h helper.h
dcc_defs.h dcc_paths.h kludge.h
/tmp/dcc-1.3.73/include # del dcc_config.h [root @ wally]
zsh: command not found: del
/tmp/dcc-1.3.73/include # rm dcc_config.h [root @ wally]
/tmp/dcc-1.3.73/include # cd .. [root @ wally]
/tmp/dcc-1.3.73 # ./configure --disable-dccm [root @ wally]
loading cache ./config.cache
Rhyolite Software DCC 1.3.73
checking for cc... cc
checking for gcc... (cached) cc
checking whether the C compiler (cc ) works... yes
checking whether the C compiler (cc ) is a cross-compiler... no
checking whether we are using GNU C... (cached) yes
checking whether cc accepts -g... (cached) yes
checking how to run the C preprocessor... (cached) cc -E
checking for pthread_mutex_lock... (cached) yes
checking for gcc __attribute__... (cached) yes
checking sizeof(long)... (cached) 4
checking for gmake... yes
checking for xargs... (cached) /usr/bin/xargs
checking for wget... (cached) /usr/bin/wget
checking for perl... (cached) /usr/bin/perl
checking for sendmail... (cached) /usr/sbin/sendmail
checking for htpasswd... (cached) /usr/bin/htpasswd
checking for rrdtool... (cached) /usr/local/bin/rrdtool
checking for sa_len... (cached) no
checking for socklen_t... (cached) yes
checking for AF_LOCAL... (cached) yes
checking for ANSI C header files... (cached) yes
checking for pid_t... (cached) yes
checking for u_int32_t... (cached) yes
checking for u_int64_t... (cached) yes
checking need for string.h... (cached) no
checking whether time.h and sys/time.h may both be included... (cached) yes
checking for strings.h... (cached) yes
checking for paths.h... (cached) yes
checking for sys/cdefs.h... (cached) yes
checking for pthread.h... (cached) yes
checking for arpa/nameser.h... (cached) no
checking for resolv.h... (cached) no
checking for utime.h... (cached) yes
checking for vsyslog... (cached) yes
checking for getipnodebyname... (cached) no
checking for getipnodebyaddr... (cached) no
checking for freehostent... (cached) no
checking for getaddrinfo... (cached) yes
checking for getnameinfo... (cached) yes
checking for freeaddrinfo... (cached) yes
checking for gai_strerror... (cached) yes
checking for hstrerror... (cached) yes
checking for _res... (cached) yes
checking for res_init... (cached) no
checking for res_query... (cached) yes
checking for dn_expand... (cached) yes
checking for inet_ntop... (cached) yes
checking for inet_aton... (cached) yes
checking for gethostid... (cached) yes
checking for localtime_r... (cached) yes
checking for gmtime_r... (cached) yes
checking for timegm... (cached) yes
checking for altzone... (cached) no
checking for futimes... (cached) yes
checking for setpgid... (cached) yes
checking for poll... (cached) yes
checking for strlcpy... (cached) no
checking for strlcat... (cached) no
checking for getprogname... (cached) no
checking for daemon... (cached) yes
checking for getifaddrs... (cached) yes
checking for freeifaddrs... (cached) yes
checking for md5.h... (cached) no
checking for sysconf(_SC_PHYS_PAGES)+sysconf(_SC_PAGESIZE)... (cached) yes
checking for __progname... (cached) yes
checking FD_SETSIZE to compute NUM_CWFS... (cached) 32
checking run directory... (cached) /var/run
checking for sysctl(kern.boottime)... (cached) no
checking for IPv6... yes
checking for Rsendto... (cached) no
creating ./config.status
.................................................c reating include/dcc_config.h
/tmp/dcc-1.3.73 # make install [root @ wally]
cc -D_FILE_OFFSET_BITS=64 -I../include -c -o aop.o aop.c
In file included from /usr/include/errno.h:36,
from ../include/dcc_defs.h:55,
from ../include/dcc_clnt.h:45,
from aop.c:42:
/usr/include/bits/errno.h:25:26: linux/errno.h: No such file or directory
In file included from /usr/include/bits/posix1_lim.h:130,
from /usr/include/limits.h:144,
from /usr/lib/gcc-lib/i486-linux/3.3.5/include/limits.h:122,
from /usr/lib/gcc-lib/i486-linux/3.3.5/include/syslimits.h:7,
from /usr/lib/gcc-lib/i486-linux/3.3.5/include/limits.h:11,
from ../include/dcc_defs.h:71,
from ../include/dcc_clnt.h:45,
from aop.c:42:
/usr/include/bits/local_lim.h:36:26: linux/limits.h: No such file or directory
In file included from /usr/include/sys/socket.h:35,
from /usr/include/netinet/in.h:24,
from /usr/include/netdb.h:28,
from ../include/dcc_defs.h:72,
from ../include/dcc_clnt.h:45,
from aop.c:42:
/usr/include/bits/socket.h:305:24: asm/socket.h: No such file or directory
In file included from ../include/dcc_defs.h:74,
from ../include/dcc_clnt.h:45,
from aop.c:42:
/usr/include/sys/param.h:23:26: linux/limits.h: No such file or directory
/usr/include/sys/param.h:24:25: linux/param.h: No such file or directory
In file included from ../include/dcc_clnt.h:45,
from aop.c:42:
../include/dcc_defs.h:186: error: `PATH_MAX' undeclared here (not in a function)
In file included from aop.c:42:
../include/dcc_clnt.h:97: error: `MAXHOSTNAMELEN' undeclared here (not in a function)
../include/dcc_clnt.h:188: error: `MAXHOSTNAMELEN' undeclared here (not in a function)
../include/dcc_clnt.h:353: error: `MAXHOSTNAMELEN' undeclared here (not in a function)
make[1]: *** [aop.o] Error 1
make: *** [all] Error 2
/tmp/dcc-1.3.73 # [root @ wally]

[juank]

etrigan63, I'm going to send you a private message. I saw your problem and I found you typed something wrong. I'll contact you directly via PM to help you.

[RichieIDI]

Juank, I noticed that untangle doesn't have pyzor installed in spamassassin. I found out it was easy to add to mine, have you thought about adding that one also?

[RichieIDI]

Also, I notice in spamassassin's debug there are tons of


[9992] info: rules: meta test OBFU_BAYES has undefined dependency 'BAYES_90'


Do you think those are bad?

[juank]

Juank, I noticed that untangle doesn't have pyzor installed in spamassassin. I found out it was easy to add to mine, have you thought about adding that one also?

I actually think it already has pyzor and razor... at least that's what I see in some logs.

How did you add it, in your local.cf? I just don't want it running twice and then give double scores which can mess up the whole scoring system.

[RichieIDI]

I actually think it already has pyzor and razor... at least that's what I see in some logs.

How did you add it, in your local.cf? I just don't want it running twice and then give double scores which can mess up the whole scoring system.

When I did spamassassin -D << foo.txt (made foo.txt) I saw pyzor executable not found
or something like that.

Also if I just typed pyzor ping (the test for pyzor) it says command not found.

Installing was just grabbing it from that same source.list you used for DCC.


Also, Forgot to mention in your code


echo "5 * * * * root /var/lib/clamav/sanesecurity.sh >> /tmp/sanesecuity.log" >>/etc/crontab


I'm sure it really doesn't matter, but the log file is missing an "r".

Also, how do those scripts work. For instance, if I look at those log files, they are blank, but I'm sure they should of run. Is there a way I can run the rules_du_jour script myself and see it go? No clue how to work bash scripts.

Once again, thanks for this. I love this software and the help people like you give!

[juank]

When I did spamassassin -D << foo.txt (made foo.txt) I saw pyzor executable not found
or something like that.

Also if I just typed pyzor ping (the test for pyzor) it says command not found.

Installing was just grabbing it from that same source.list you used for DCC.


Also, Forgot to mention in your code


echo "5 * * * * root /var/lib/clamav/sanesecurity.sh >> /tmp/sanesecuity.log" >>/etc/crontab


I'm sure it really doesn't matter, but the log file is missing an "r".

Also, how do those scripts work. For instance, if I look at those log files, they are blank, but I'm sure they should of run. Is there a way I can run the rules_du_jour script myself and see it go? No clue how to work bash scripts.

Once again, thanks for this. I love this software and the help people like you give!

First, thanks for the "r" typo. It doesn't really matter, it's just a log.
As far for the rules_du_jour, you can run it by hand, just do:

/usr/share/spamassassin/rules_du_jour.bash

So, after you installed PYZOR, how do you know your Untangle SA is using it?

[RichieIDI]

First, thanks for the "r" typo. It doesn't really matter, it's just a log.
As far for the rules_du_jour, you can run it by hand, just do:

/usr/share/spamassassin/rules_du_jour.bash

So, after you installed PYZOR, how do you know your Untangle SA is using it?

Well, I have no clue if untangle is using it :) But, when I run that same spamassassin -D << sample-spam.txt it says


[25203] dbg: pyzor: pyzor is available: /usr/bin/pyzor
[25203] dbg: info: entering helper-app run mode
[25203] dbg: pyzor: opening pipe: /usr/bin/pyzor check < /tmp/.spamassassin25203g8CmT9tmp
[25718] dbg: util: setuid: ruid=0 euid=0
[25203] dbg: pyzor: got response: 82.94.255.100:24441 (200, 'OK') 2 0


Oh, I got sample-spam.txt from some website I found while googling.


wget http://www200.pair.com/mecham/spam/sample-spam.txt


I guess you can also do spamassassin -D pyzor < sample-spam.txt and it kills a bunch of the mumbo-jumbo.

[juank]

Thanks RichieIDI,

I just modified my script here and I'm going to test it for a day to see if pyzor actually runs as is supposed to do it.

I'll post my new script as soon as my live test ends.

Thanks for your input/feedback!

[RichieIDI]

Thanks RichieIDI,

I just modified my script here and I'm going to test it for a day to see if pyzor actually runs as is supposed to do it.

I'll post my new script as soon as my live test ends.

Thanks for your input/feedback!

Thank you for starting this thread!

Oh, another thing I was just reading about, I guess we shouldn't use the rules_du_jour.bash script anymore, should use sa-update.

http://daryl.dostech.ca/sa-update/sare/sare-sa-update-howto.txt

[juank]

Thank you for starting this thread!

Oh, another thing I was just reading about, I guess we shouldn't use the rules_du_jour.bash script anymore, should use sa-update.

http://daryl.dostech.ca/sa-update/sare/sare-sa-update-howto.txt

Yes, that's on my ver 2.0 script. I've been testing it for 3 weeks, but i don't want to post it yet.

Now i know I have a friend to help me test it.

I'll PM you so we can get in contact directly. Thanks

[TonyG]

>>>Yes, that's on my ver 2.0 script. I've been testing it for 3 weeks, but i don't want to post it yet.<<<

Let us know when it's ready!

Thanks,


TonyG

[dmorris]

>>>Yes, that's on my ver 2.0 script. I've been testing it for 3 weeks, but i don't want to post it yet.<<<

Let us know when it's ready!

Thanks,

TonyG

Tony - Is that you? :D
I smell trouble!

sorry for thread hijack

[juank]

Tony - Is that you? :D
I smell trouble!

sorry for thread hijack

Now I'm scared ! :eek: Who is Tony?

[etrigan63]

Well after reinstalling my Untangle server i was able to successfully execute your script. Now we'll see how it helps...

[djws44]

To give an update - we broke this down into several separate bugs currently to be put into 5.1

If all goes well in QA - juank's changes will be released in Thunderbird! :D

http://bugzilla.untangle.com/show_bug.cgi?id=3308
http://bugzilla.untangle.com/show_bug.cgi?id=3310
http://bugzilla.untangle.com/show_bug.cgi?id=3317

DMorris, What is the ETA of 5.1? When will this upgraded spam feature be available without manual entry of the script??

[mdh]

Current plan for 5.1 is late January.

[juank]

if anybody out there is using my first script, please let me know (via a PM) and I'll tell you how to "upgrade it". After that, I'm going to publish the new script (ver 1.2).


Changes:
1. Use SA-UPDATE to update SARE rules from RulesEmporium.
2. Installation and use of PYZOR for the Spam Blocker



Thanks.

[Pete]

I am!! :) Sent you a Pm

[nomad]

The Spam blocker script worked great. My spam has drop 60% since I installed it.

Thanks...

[juank]

Thanks a lot for your comments. I had a new updated script I was going to post last week, but I decided to wait for the 5.1 release (late Jan 2008 ... maybe...).

[spg]

There is not a version of MailEssentials and MailSecurity from GFI for Exchange 2007 at present.

Word from the grape Vine is that GFI will never release it.... dont expect it (or use it, its a god awful pos)

[TonyG]

>>>Current plan for 5.1 is late January.<<<


We untangle minions stand waiting...


TonyG

[juank]

Don't worry, is coming... we're actually testing a DEV release... (not ready for show yet).

[ali_shaki]

I can not install build-essential
I am having a dependence error.
how can I install build-essential please help me
-----------------------------
apt-get -f install -y build-essential [root @ odsur]
Reading Package Lists... Done
Building Dependency Tree... Done
You might want to run `apt-get -f install' to correct these:
The following packages have unmet dependencies:
build-essential: Depends: libc6-dev but it is not going to be installed or
libc-dev
Depends: gcc (>= 3:3.3) but it is not going to be installed
Depends: g++ (>= 3:3.3) but it is not going to be installed
Depends: make but it is not going to be installed
Depends: dpkg-dev (>= 1.4.1.19) but it is not going to be installed
knx-installer: Depends: initrd-tools but it is not going to be installed
libdb4.4: Depends: libc6 (>= 2.3.6-6) but 2.3.2.ds1-22sarge6 is to be installed
libgdbm3: Depends: libc6 (>= 2.3.6-6) but 2.3.2.ds1-22sarge6 is to be installed
libhtml-parser-perl: Depends: perlapi-5.8.8 but it is not installable
Depends: perl (>= 5.8.8-6) but 5.8.4-8sarge5 is to be installed
Depends: libc6 (>= 2.3.6-6) but 2.3.2.ds1-22sarge6 is to be installed
libperl5.8: Depends: libc6 (>= 2.3.6-6) but 2.3.2.ds1-22sarge6 is to be installed
Depends: perl-base (= 5.8.8-7etch1) but 5.8.4-8sarge5 is to be installed
locales: Depends: glibc-2.3.6.ds1-1 but it is not installable
perl-modules: Depends: perl (>= 5.8.8-1) but 5.8.4-8sarge5 is to be installed
perl-suid: Depends: perl (= 5.8.8-7etch1) but 5.8.4-8sarge5 is to be installed
Depends: libc6 (>= 2.3.6-6) but 2.3.2.ds1-22sarge6 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
/usr/share/spamassassin #
-----------------------------

[shawnbishop]

I am going to give it a go, once I can get the Untangle Server to work properly...

Thanks

[Stanna]

juank, Is the script on the front page your 1.2 version - it still seems to have the bash script to update rules_du_jour. Since it is now near end Feb, is it worth you putting your version 1.2 script up so people don't have to keep asking "When is 5.1 going to be released".
Thanks

[Stanna]

OK - thought I'd give this version of the script a go - all seemed to be going well up until:

make install

Stuff seemed to be happening but then stopped at some lines saying:

else\
sh -e fix-map -c ../cdcc/cdcc -h /var/dcc;\
fi

I get left with a solid cursor on the line after fi
Eventually I hit Ctrl C and it then dropped back to the prompt and I could continue with the rest of the script ( I did get an error come up saying server not found when trying cdcc "delete 127.0.0.1"

Is there something else I should try?

Thanks

Simon

[mfmjos]

I was hoping to get a status of the SPAM blocker improvement script. I'd like to implement this but would like to use the most current version. Also did I hear/read or make up the thought of having this incorporated in 5.1?

[juank]

Most of them are incorporated in 5.1 but I'm working on the script to publish here to whoever wants to try it in 5.1

[Rob]

I seem to have missed quite a bit here. I've just got my Untangle server running now. Version 5.1. :)

What I am worried about though is the spam in the Internet cafe. So, Jaunk I take it the spam filter you made is already in 5.1 right on the first post?

What are you making for 5.1 now? A new one?

Would love to give it ago and post back my results :)

PM me if you want.

Thank you.

[Ron Chandy]

Juank I am impressed the type of work you are doing I was wondering if this spam filter upgrade also help contoll pop3 type of spam ?? As I know only SMTP can be controlled the rest like POP3, IMAP can only be marked ??

[juank]

Sorry, POP3, IMAP can only be marked, it is the nature of their protocol.

[Elliven]

Dont know if this is the right place to post this:

http://spamgrid.ai-applied.nl

I quote:

...your e-mail client will connect to SpamGrid, which will in turn connect to your e-mail server. When e-mail messages are retrieved from the server, SpamGrid will send a non-reversible encoding of those e-mail messages to the SpamGrid server.

By analysing those encodings, the server will determine whether the e-mail distribution is a mass e-mail distribution. Once a distribution has been characterized as a mass distribution, it's geographical distribution is analysed. According to these two factors, a message (and subsequent messages of the same distribution) can be characterised as spam.

This implies that unlike with other spam filters, normal, personal, e-mail messages can't be characterised as spam*, since they don't conform any of the two necessary rules. Also, newsletters conform to only one of the two rules, thereby making it possible for newsletters to pass the filter as well, while all spam is flagged as being spam

I'm no AntiSpam fundi, but it seems to me that this is a much better way of blocking Spam than anything anyone has come up with so far?
Perhaps this can be built into Untangle??
:popcorn:

[OCMnetworks]

Juank,

Fab job with this script -IMO the spam filtering was/ is one of the few things letting the Untangle down
Do you know how much of this script made it into the 5.10 release? I've got a customer running it who is still experiencing large volumes of spam getting through

Is the script on the first page ok to run on 5.10?

TIA

Ben

[juank]

Hello,
Please don't use any of my scripts in 5.1. Most of the stuff is already included in 5.1 and there are other scripts I'm working on that may be included later by the Untangle team.
Thanks