Blue Flower

Monday, 06 July 2015 01:30

How to teach Bayes your users Spams

Written by

This doc will show you how to scan each users .Spam folder, teach them as spam and/or ham and then delete them within x days

First we will want to download the spamlearn.sh to a bin folder. I usually put all my scripts in ~root/bin but you
can put them anywhere you want:


# cd ~root/bin
# fetch http://www.goodcleanemail.com/files/spamlearn.sh
# vi spamlearn.sh

The first 2 settings SAPROG and SAFLAGS should be ok the way they are provided you're using FreeBSD.
The DOMAIN_BASE_PATH is the default path of vpopmail.
The DOMAINS setting is A list of your local domains which you would like to use this script to learn against. Make sure the
Domains are seperated by spaces!
The next three settings are if you have a catchall setup. If you do, go ahead and uncomment them and set them as you see fit.
The next three settings are if you quarantine spam. If you do, go ahead and uncomment them and set them as you see fit.
CHECK_USERS is just how you tell Bayes to train. I set this to 2 myself. I never teach my system hams.
USER_SPAM_DIRS and USER_HAM_DIRS is the default setting. Do not change unless you changed the skel setup.
DELETE_USER_SPAM and DELETE_SPAM are optional. As it is set right now, it is set to delete spams older than 30 days.
The last 2 setttings are optional. Depends on how you want the output of the script changes. I would leave them as they are
unless you are having problems.

Now to set the correct permissions and then run it:


# chmod 755 spamlearn.sh
# ./spamlearn.sh

If you are comfortable with the way it runs, Go ahead and put it into cron (man 5 crontab for more info) and you should be all set!

Monday, 06 July 2015 00:35

Optimizing qmail/SpamAssassin to catch more Spams

Written by

When you first install SpamAssassin from either source, rpm or Ports, The default setup for SpamAssassin still allows a lot of spams to still get through. The following are a few recommended things to use to help SpamAssassin filter more spams out for you.

If you happen to get a lot of spams to invalid users, I would highly suggest taking a look at john Simpsons validrcptto patch. This is included with his combined patch as well. If you use his patch, you will also need to replace the original qmail-smtpd/run script with Johns as it breaks smtp-auth. Take a look at the following URLs

http://qmail.jms1.net/patches/
http://qmail.jms1.net/scripts/

If you would like to enable jgreylist, follow this website:

http://qmail.jms1.net/scripts/jgreylist.shtml

The next thing I would recommend is to enable the Bayes Database. Take a look at:

http://freebsdrocks.net/index.php/13-useful-qmail-utilities/61-how-to-enable-bayes-autolearning

 I would suggest adding rules to SpamAssassin to mark up messages better. The only thing I have found about some of these rules is that when you run spamassassin --lint, You do get some errors from time to time.

http://freebsdrocks.net/index.php/16-useful-qmail-utilities/spamassassin/58-how-to-add-rules-to-spamassassin 

You can also enable razor/pryzor here:

http://freebsdrocks.net/index.php/13-useful-qmail-utilities/52-installing-and-configuring-razor-and-pryzor

Another good thing to use is SpamAssassin Auto-Learning with Site-Wide Bayes and User Feedback. This will allow your users to send spams or hams via forwarding as an attachment to report them to bayes as ham or spams.

http://freebsdrocks.net/index.php/16-useful-qmail-utilities/spamassassin/59-spamassassin-auto-learning-with-site-wide-bayes-and-user-feedback

If you implemented the site-wide bayes with feeedback, adding the Squirrelmail spam or ham reporting option makes reporting via Squirrelmail really simplistic.

http://freebsdrocks.net/index.php/13-useful-qmail-utilities/47-setting-up-ham-and-spam-buttons-for-squirrelmail

There are some new spams going around that are gifs. Here is a link on how to get rid of those types of spams:

http://freebsdrocks.net/index.php/16-useful-qmail-utilities/spamassassin/57-fuzzyocr-walkthrough

The object of this howto is getting your SpamAssassin 3.0.x Bayes Database
effective system-wide and allow your users to feed mis-tagged spam back to the
server where a script automatically runs sa-learn on it.  In order to use this
method you need the following:

A properly working email server with Spamassassin 3.0.x
RipMIME from http://www.pldaniels.com/ripmime
You can install the FreeBSD port: /usr/ports/mail/ripmime
An email account on your server (i.e. This email address is being protected from spambots. You need JavaScript enabled to view it. )
for the users to send the spam to
The learnspam script included in this package 
Users must send the spam emails as ATTACHMENTS to your thisisspam email address

This howto is based on a qmail server setup according to http://www.qmailrocks.org
Other servers will be similar but you must adjust directories and accounts accordingly.

Please note: If you are running the freebsdrocks setup, you do not need to change the spamd service to run as a different user. It is already running as user qscand.

STEP 1 - The System Account:

The System-wide Bayes Database and spamassassin need to operate as the same user.
Normally that would be spamd as set in /etc/sysconfig/spamassin (or similar)
But the Autolearn script must be able to R/W the mail directories on the server
and the Bayes Database.  Spamd cannot R/W mail directories so you must run the
script as either root (cron.daily) or vpopmail.  However, Vpopmail does not have
R/W permissions to the Bayes Database if spamd is running spamassassin. 
For those who do not wish to risk running the script as root, simply change
the spamd user to qscand by setting the -u and -h options in
/etc/sysconfig/spamassin from spamd to qscand.
Then when you restart spamassassin, ps aux should show spamd running as
qscnad who is able to R/W the Bayes Directory. 

Once you decide which account
will run spamassassin and the autolearn script, choose where in that account's
home directory to put the database, the default is
/home/(account name)/.spamassassin

STEP 2 - Setting up Bayes and Autolearning in Spamassassin:

Edit /usr/local/etc/mail/spamassassin/local.cf and insert the following lines:

bayes_path /path/to/your/bayes/directory ( as you chose in Step 1)
use_bayes 1
bayes_auto_learn 1

Save the file and restart spamassassin
Run sa-learn --sync to resync the database
Run sa-learn --dump magic and you should see nham and nspam at 0

You need 200 ham and 200 spam in your database for Bayes to autolearn.
If you have good emails in your users' /cur directories do the following:

# find /home/vpopmail/domains -type d -name cur -exec sa-learn --nosync --ham {}/* ;

Then run sa-learn --sync and sa-learn --dump magic to see that they are there.
Otherwise gather some legit email from your users or other sources into a
directory on the server and run sa-learn --nosync --ham on them, then --sync again.

Find some spam to force feed the database - drop it into a folder and run
#sa-learn --nosync --spam /path/to/spam/*
Then run sa-learn --sync and sa-learn --dump magic again
to make sure the database is growing.  You should see numbers climbing steadily
as spamassassin automatically learns spam and ham as mail flows through the server.


STEP 3 - Setting up the Feedback Autolearn Script

After setting up your spam account and installing RipMIME,
Edit the learnspam script variables per your preferences and system.

The system account the script runs as must have /usr/local/bin in their $PATH to find
RipMIME. If you chose to run the script as root (from cron.daily) you will need to
insert this line in the script:   PATH="$PATH":/usr/local/bin
Remember, however, that running anything as root has risks - do so at your own risk.
Forward some spam email to the thisisspam account and run the script to test it. 
Make sure that the logfile shows that the emails were RipMIME'd and that they were
learned by sa-learn.  If sa-learn has seen them before it will not learn them again
unless it forgets them first, so do not be suprised it you see more examined
than learned. Once the script is tested, enter the cron job for it and watch your
logs for activity.

Maintenance - LogRotate does a fine job of rotating the logs on the system.  A recommended
entry for the salearn.log is:

# AutoLearn Spam Log
# This should rotate the log every week
# and keep one month's worth of logs archived
/var/log/salearn.log {
weekly
rotate 4
nocompress
}


You can download the following related files:

salearn.sh and salearn.log
http://www.goodcleanemail.com/files/tarballs/salearn.tgz

References:

http://freebsdrocks.net/index.php/13-useful-qmail-utilities/50-forwarding-emails-as-attachments-in-ms-outlook

Monday, 06 July 2015 00:34

How to add rules to SpamAssassin

Written by

This page has been updated 7/18/2016.

These are additional rulesets you can add to SpamAssassin to help improve spam detection. Please review each rule before you implement this. This walkthrough is optimizied for FreeBSD 10.2 RELEASE or STABLE.

Please note: I have removed the sa-blacklist rules as they can cause memory issues. Please see http://wiki.apache.org/spamassassin/OutOfMemoryProblems

First we need to choose a backup path. This path should reside outside the standard spamassasssin directory. Lets use /usr/local/etc/mail/spambackup


# mkdir /usr/local/etc/mail/spambackup

Now lets copy all the files that are in the /usr/local/etc/mail/spamassassin folder. We want to preserve the file history in this case.


# cd /usr/local/etc/mail/spambackup
# cp -Rp /usr/local/etc/mail/spamassassin/ /usr/local/etc/mail/spambackup

We will add rules that are no longer updated and then we will add a script to update additional rules. First we need to fetch the rules along with the cfupdates script:


# mkdir ~root/rules
# cd ~root/rules
# fetch http://freebsdrocks.net/qmail2/cfupdates2.tgz
# tar zxvf cfupdates2.tgz
# rm cfupdates2.tgz

Now lets copy over the non-updated rules to the spamassassin folder first:


# cp -Rp ~root/rules/*.cf /usr/local/etc/mail/spamassassin
# rm *.cf

Now you will want to copy the cfupdates to a bin folder. I keep mine in ~root/bin so lets use that. Feel free to use any path you choose.


# mkdir ~root/bin
# cp ~root/rules/cfupdates.sh ~root/bin
# cd ~root/bin
# chmod 755 cfupdates.sh

Here is my crontab entry to run at 3AM every day. It is not advised to run this script than any more than once in a 24 hour period.

0 3,15 * * * ~root/bin/cfupdates.sh > /dev/null 2>&1

Additional notes:

If you are using bayes this script will also backup your bayes folder. Run the following first


# mkdir /usr/local/etc/mail/spambackup/.spamassassin

Uncomment the two bayes sections in cfupdates.sh

You can also add your own rules. I provided stock.cf as an example. Take a look at the following:

body WEEKLY_STOCK_SPAMS /\bWeekly Stock Report\b/i
score WEEKLY_STOCK_SPAMS 10.0
describe WEEKLY_STOCK_SPAMS     This is a Stock Spam

The first section you need to give a description which is part of the score and describe. This needs to be the same in all 3 sections. The next part in the body /\bWeekly Stock Report\b/i
is what you want to "tag" for spamassassin to see.
The score is what you need to determine. If you're using sa-delete please take this into account.
The last section is just the description of the rule. That's all there is to scoring a message.

You can also additional rules from the /usr/ports/mail/spamass-rules port. Just run make installl clean.

More rules are located here: http://spameatingmonkey.com/usage.html

Special thanks to Steve Donohue

Monday, 06 July 2015 00:34

FuzzyOCR Walkthrough

Written by

FreeBSD FuzzyOCR SA Plugin for FreeBSD

Required ports to install are netpbm, gocr, imagemagick, giflib and the String::Approx Perl module.

# cd /usr/ports/graphics/netpbm && make install clean
# cd /usr/ports/graphics/gocr && make install clean
# cd /usr/ports/graphics/ImageMagick && make install clean
# cd /usr/ports/devel/p5-String-Approx && make install clean
# cd /usr/local/etc/mail/spamassassin
# fetch http://users.own-hero.net/~decoder/fuzzyocr/fuzzyocr-latest.tar.gz
# tar zxvf fuzzyocr-latest.tar.gz
# cd FuzzyOcr-version

edit FuzzyOcr.cf and change all "/etc/mail/spamassassin/" to "/usr/local/etc/mail/spamassassin/"

I set my focr_logfile to /var/log/FuzzyOcr.log

also edit the FuzzyOcr.pm file.  Search for "$logfile", and you will notice a line calling the log file again.  I just pointed it to the same location.  Not sure why it's called twice.

Now we finish up.

also in the FuzzyOcr.cf file you will need to change the paths of the "Helper Applications" located around line 41.  Change them to the following unless you installed them to /usr/bin/.

focr_bin_giffix /usr/local/bin/giffix
focr_bin_giftext /usr/local/bin/giftext
focr_bin_gifasm /usr/local/bin/gifasm
focr_bin_gifinter /usr/local/bin/gifinter
focr_bin_giftopnm /usr/local/bin/giftopnm
focr_bin_jpegtopnm /usr/local/bin/jpegtopnm
focr_bin_pngtopnm /usr/local/bin/pngtopnm
focr_bin_ppmhist /usr/local/bin/ppmhist
focr_bin_convert /usr/local/bin/convert
focr_bin_identify /usr/local/bin/identify
focr_bin_gocr /usr/local/bin/gocr

Be sure they are all uncommented.

# cp FuzzyOcr.* /usr/local/etc/mail/spamassassin/
# cd /usr/local/etc/mail/spamassassin/
# mv FuzzyOcr.words.sample FuzzyOcr.words
# /usr/local/etc/rc.d/sa-spamd.sh restart

*Notice*
If you are using w0ls0n's cfupdates script, you should remove the rm *.* or otherwise your Fuzzy confs will go bye bye.

#* Writen By mintee 10/17/2007 *

Monday, 06 July 2015 00:33

How to get a "SpamAssasin Stats" report

Written by

Updated 10-18-05

Download the following script below:

# wget http://www.goodcleanemail.com/files/tarballs/sastats.tgz

# chmod 755 sa-stats.pl

Run it and you will get a nice output :-)

Now add it to cron like so:

0 1 * * * /path/to/sa-stats.pl

This will send that report to root@hostname at 1A

Page 8 of 23