How to convert a MT box in an Anti Spam server with v2.9

By admin

Diambil dari http://forum.mikrotik.com/viewtopic.php?t=15721

Hi there, for months I have the idea how to transform one Mikrotik OS in an Anti Spam server. After more than 20 days applying few scripts, firewall rules and address list, I have achieved reduce from 45.000 mails per day to only 11.000/12.000 without many complaints from my customers.

Before continuing, some details about this:

Yes, I know that exists others solutions.
Yes, I know with Linux can obtain the same results.
Yes, I know that it seems a crazy solution.
Yes, I know (in the practice) that this solutions generate a moderate cpu usage. I have a Pentium IV with 75% of cpu usage (this can change with new features from MT… see scripts explanation) and we are a little ISP.
Yes, yes, yes…
but
I use MT from six years ago (when John Tully & Arnis Riekstins answered the company mails themselves) and always I try to resolve any networking necessity whit MT.
I had the trust that can resolve this whit MT.
and after thinking about this, I can’t never sleep all night from many days, so, for my health and wife I made it !!!


The scripts:

I use 3 scripts:

itwbspam (run every second)
spamtocheck (run every 10 seconds)
tempclear (run every 30 minutes)

Script itwbspam (remove the # lines to work, I comment the most important parts):

:set SCRIPT itwbspam;
# Always I think that MT permit one script running at the same time, for my surprise is not, so I
# use this to run at the same time one script (simulating a threads).
# You can reduce this, if your cpu usage is high or your mail traffic is reduced.
# In my case, less than 10 permit detect if the incoming SMTP connection is an spam or not
# quickly, and the cpu usage is reasonable.

# If you need run only one script at the same time, set at < 1 because the script it's count the
# same one when running.

:if ([:len [/system script job find script=$SCRIPT]] < 10) do={ \

# I use a firewall rule to add ips to this list (see below in firewall rules).

:set ALIST checkspam;

:foreach IPT in [/ip firewall address-list find list=$ALIST] do={ \

:set COMM [/ip firewall address-list get $IPT comment];

# For not process the same ip in more than one script, I put a comment to prevent this.

# Maybe in a future we can put a comment in a DYNAMIC list (I don't know why we can do that)

# so that can reduce the cpu usage.

:if ($COMM = "") do={ \

/ip firewall address-list set $IPT comment=checking;

# Now you need to revert the ip for querying a dnsbl server and know if the server that try to

# connect to your SMTP server is a spam or not.

:set IP [/ip firewall address-list get $IPT address]; :set IPL $IP;

:set l [:len $IP]; :set n [:find $IP .];

:set A [:pick $IP 0 $n];

:set IP [:pick $IP ($n + 1) 15]; :set n [:find $IP .];

:set B [:pick $IP 0 $n];

:set l [:len $IP]; :set IP [:pick $IP ($n + 1) 15]; :set n [:find $IP .];

:set C [:pick $IP 0 $n];

:set l [:len $IP]; :set IP [:pick $IP ($n + 1) 15];

:set D $IP;

:set COUNT 0;

# Now ask to black lists.

:set DSBL .dnsbl.sorbs.net;

:set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];

:if ($IP != "0.0.0.0") do={ :set COUNT ($COUNT + 1); }

:set DSBL .blackholes.five-ten-sg.com;

:set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];

:if ($IP != "0.0.0.0") do={ :set COUNT ($COUNT + 1); }

:set DSBL .dnsbl.tqmcube.com;

:set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];

:if ($IP != "0.0.0.0") do={ :set COUNT ($COUNT + 1); }

:set DSBL .zen.spamhaus.org; :set SPAM 127.0.0.4;

:set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];

:if ($IP = $SPAM) do={ :set COUNT ($COUNT + 1); }

:set DSBL .bl.spamcop.net; :set SPAM 127.0.0.2;

:set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];

:if ($IP = $SPAM) do={ :set COUNT ($COUNT + 1); }

# You can add so many black lists as you want, only you need set the DSBL and SPAM variable.

# If the ip returned from the dnsbl server coincides whit SPAM, the COUNT variable add

# by one.

# From servers like sorbs.net or five-ten-sg that return more than one value, I don't use the SPAM

# variable and check only the value if different to 0.0.0.0 and I assuming is spam when is

# different.

:if ($COUNT < 3) do= { \

# If the ip server COUNT lees than 3, I assuming maybe need more checks, so you can add

# others black lists to be safe that this ip it's not a spam server !.

:set DSBL .list.dsbl.org; :set SPAM 127.0.0.2;

:set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];

:if ($IP = $SPAM) do={ :set COUNT ($COUNT + 1); }

:set DSBL .dnsbl-1.uceprotect.net; :set SPAM 127.0.0.2;

:set IP [:resolve ($D . "." . $C . "." . $B . "." . $A . $DSBL)];

:if ($IP = $SPAM) do={ :set COUNT ($COUNT + 1); }

}

# Now according to the value in COUNT you can decide which is the address list that  it

# corresponds to each case. In my case this is the best solution:

# if the server COUNT 1, is can or not a spam server, so it's fine to me and pass,

# if have 2, I think they are amateurs and they need to sample with other ISPs,

# if have 3 or more... hey, leave me alone !!!

:if ($COUNT > 1) do= { \

:if ($COUNT = 2) do={ /ip firewall address-list add list=tempspamlight address=$IPL; }

:if ($COUNT = 3) do={ /ip firewall address-list add list=tempspammedium address=$IPL; }

:if ($COUNT > 3) do={ /ip firewall address-list add list=tempspamhard address=$IPL; }

# Yes, you need to use STATIC list because we can't add a DYNAMIC list from script or

# timeout in a STATIC list (I don't know why, if we can in the future, this reduce considerably

# the cpu usage) and maybe, this server it's hacking so you can't assume that this server

# is a spam server only for a while, so you need to add a timeout value (see below in

# firewall rules).

} else= { \

/ip firewall address-list add list=tempsmtp address=$IPL;

}

# Once the server is cataloged you need delete the ip from the temp list

/ip firewall address-list remove $IPT;

}

}

}

You need this script running all time to check each ip in the list if are spam server or not. After that you have 3 different categories (in this example, you can add more than 3 if you like) so only you need transform this STATIC list in DYNAMIC list whit a timeout value accord to the type of spam that the script it detecting (see below in firewall rules). In my case I have more than 20.000 ip address counting white lists and black lists.

Script spamtocheck: this script purpose it’s only transform a DYNAMIC list to STATIC list because in DYNAMIC list you can’t put a comment (why !!!???) or counter, or anything that can I use to prevent the script itwbspam take an ip that is checking in another script itwbspam at the same time.

:set ALIST itwbspam; :set BLIST checkspam;:local IPLIST [/ip firewall address-list find dynamic=yes list=$ALIST];:foreach IP in $IPLIST do={ \:set IPL [/ip firewall address-list get $IP address];

/ip firewall address-list add list=$BLIST address=$IPL;

/ip firewall address-list remove $IP;

}

Script tempclear: thescript purpose it’s only clear the address list of servers that never try to contact again our SMTP server. If in the future we can add a timeout to STATIC list, this script is not necessary any more and YES maybe reduce the cpu usage.

:set ALIST tempsmtp;:for each IP in [/ip firewall address-list find list=$ALIST] do={ /ip firewall address-list remove $IP; }:set ALIST tempspamlight;:for each IP in [/ip firewall address-list find list=$ALIST] do={ /ip firewall address-list remove $IP; }

chain=smtp protocol=tcp src-port=25 src-address-list=staticwhitelist action=accept

:set ALIST tempspammedium;

:for each IP in [/ip firewall address-list find list=$ALIST] do={ /ip firewall address-list remove $IP; }

:set ALIST tempspamhard;

:for each IP in [/ip firewall address-list find list=$ALIST] do={ /ip firewall address-list remove $IP; }

Now you need to add some firewall rules to make this work.

Firewall rules:

I have an separate server before my SMTP server, so I put this rules in filter rules/XXX chain, maybe you need change this according to your configuration. Obviously I have more than this rules, I mention here only it related whit this scripts.

First you need to be safe that your customers can arrive to the server. In my case I have a relay server configured in different ip for the SMTP in the same server, but if you use the same ip server to receive and send emails you need this rule.
IMPORTANT, if you have customers out side your network whit this scripts they can’t send mails from the SMTP server any more unless if your SMTP server are connecting to internet whit the same box where you run this scripts. In this case you need to add a few mangle rules to know which connection it’s established for your SMTP server and mark the packet to pass over the spam verification.

;;; Accept network customers !!!chain=smtp protocol=tcp dst-port=25 src-address-list=customers action=accept

Now you need to minimize false positive, for that reason I have a bash script (see below in bash script) that collecting the information from two white list and put in STATIC list (whitelist). I cron this bash to run each 3 hours and accept any mail connection that come from this servers. Whit this you can be peaceful about hotmail, yahoo, gmail and many others ISPs, Companies, etc.
On the other hand, I have a static white list address from ISPs or Companies that I know don’t make spam or are big ISP and they SMTP servers appears in more than one black list, so I need this servers pass over the spam verification (staticwhitelist). For example, here in Argentina, one of the biggest ISP is Speedy and their SMTP servers appears in more than one black list.

;;; Accept white list !!!chain=smtp protocol=tcp dst-port=25 src-address-list=whitelist action=accept

chain=smtp protocol=tcp dst-port=25 src-address-list=staticwhitelist action=accept

chain=smtp protocol=tcp src-port=25 src-address-list=whitelist action=accept

chain=smtp protocol=tcp src-port=25 src-address-list=staticwhitelist action=accept

Whit this rule, you transform all ips that pass the spam verification whit the script itwbspam into DYNAMIC ip, with timeout according to your experience. In my case 12 hours is fine to verify again if are spammer or not.

;;; Pass the temp to smtpokchain=smtp src-address-list=tempsmtp action=add-src-to-address-list address-list=smtpok address-list-timeout=12h

Now you are sure that the ip server in the smtpok is not a spam server or is a light spammer (COUNT = 1 in my case), so you accept the connection.

;;; Accept tempsmtp !!!chain=smtp protocol=tcp dst-port=25 src-address-list=smtpok action=accept

chain=smtp protocol=tcp src-port=25 src-address-list=smtpok action=accept;;; NOT accept static spam !!

chain=smtp protocol=tcp dst-port=25 src-address-list=staticspam action=reject reject-with=icmp-network-unreachable

Now you need to pass from STATIC list to DYNAMIC list all that servers ips that they are spam and put the timeout according to each categories.

;;; Pass the temp to spamchain=smtp src-address-list=tempspamlight action=add-src-to-address-list address-list=spamlight address-list-timeout=2h

chain=smtp src-address-list=tempspammedium action=add-src-to-address-list address-list=spammedium address-list-timeout=6h

chain=smtp src-address-list=tempspamhard action=add-src-to-address-list address-list=spamhard address-list-timeout=2d

The idea here is reject the packets from ips in the temp address lists that at this time is not checked from our script. Off course the first mail is not delivery to the SMTP server, but if the server that try to connect is not spammer, the most probable thing it’s thats try to connect again later (few minutes last in my experience) in contrast with spammers servers that it never come back again. After that, reject all the spam lists.

;;; Pass the temp to spamchain=smtp src-address-list=tempspamlight action=add-src-to-address-list address-list=spamlight address-list-timeout=2h

chain=smtp src-address-list=tempspammedium action=add-src-to-address-list address-list=spammedium address-list-timeout=6h

chain=smtp src-address-list=tempspamhard action=add-src-to-address-list address-list=spamhard address-list-timeout=2d;;; Not accept temps

chain=smtp protocol=tcp dst-port=25 src-address-list=itwbspam action=reject reject-with=icmp-host-unreachable

chain=smtp protocol=tcp dst-port=25 src-address-list=checkspam action=reject reject-with=icmp-host-unreachable

chain=smtp protocol=tcp dst-port=25 src-address-list=tempsmtp action=reject reject-with=icmp-host-unreachable

;;; NOT accept spamhard !!

chain=smtp protocol=tcp dst-port=25 src-address-list=spamhard action=reject reject-with=icmp-network-unreachable

;;; NOT accept spammedium !!

chain=smtp protocol=tcp dst-port=25 src-address-list=spammedium action=reject reject-with=icmp-network-unreachable

;;; NOT accept spamlight !!

chain=smtp protocol=tcp dst-port=25 src-address-list=spamlight action=reject reject-with=icmp-network-unreachable

The last thing that you need to do is add a rule to fill the address list spamcheck whit new connections.

;;; Add to checkchain=smtp protocol=tcp dst-port=25 tcp-flags=syn connection-state=new src-address-list=!checkspam action=add-src-to-address-list address-list=itwbspam  address-list-timeout=20m

chain=smtp protocol=tcp dst-port=25 tcp-flags=syn connection-state=new src-address-list=!checkspam action=reject reject-with=icmp-host-unreachable

chain=smtp protocol=tcp src-port=25 connection-state=established action=log log-prefix=""

That’s all !!!

Basically the process can resume like this:

One server try to connect to our SMTP server
If the ip is in the white list address the packet is delivery to our server
If the ip server is in the address list smtpok the packet is delivery to our server
If the packet arrives to the rule that check is a new connection, the ip is added to the address list itwbspam
The script spamtocheck run every 10 seconds and move the ip address in the DYNAMIC list itwbspam to a STATIC list checkspam because we can’t add a comment in DYNAMIC lists !!
The script itwbspam run every 1 second and check every ip address in the STATIC list checkspam to determine this ip is a spam server or not, if is spam then go to a tempspam list categorie, is not go to tempsmtp

Bash script: inspired by one on this forum.

#!/bin/bash

#

today=`date "+%m%d%y"`;

workdir="/home/mt";

targets=(you ip or ips server);cd $workdir

echo > wlists.rsc

# whitelist

echo :foreach subnet in [/ip firewall address-list find list=whitelist] do=\{ /ip firewall address-list remove \$subnet \} >> wlists.rsc

wget -t 2 -O dnswl.txt http://www.dnswl.org/data/generic-dnswl

egrep "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" dnswl.txt | awk '{print $1}' | awk -F";" '{print $1}' | grep -v ";" | sort +1 | awk '{print "/ip firewall address-list add l

ist=whitelist address=" $1}' >> wlists.rsc

wget -t 2 -O rediris.txt http://www.rediris.es/abuses/eswl/data/mtawl.postfix

egrep "[0-9]{1,3}" rediris.txt | awk '$1 > 0 " " $1 {print "/ip firewall address-list add list=whitelist address="$1"/32"}' >> wlists.rsc

echo >> wlists.rsc

# Remove duplicate lines

sort wlists.rsc | uniq > wlistsu.rsc

# Reduce to small files to be more manageable for MT

split wlistsu.rsc -a 5 -d -l 500 spwlists

# foreach target($targets)

for X in ${targets[@]}; do

  echo $X;

    for I in `find -name "spwlists?????"`; do

    mv $I "$I.rsc";

    echo "put $I.rsc" | ftp $X 21

  done

  for I in `find /home/mt/ -name "spwlists?????.rsc" -printf "%f\n" | sort`; do

    echo "procesando $I";

    ssh -l black -i /home/mt/black.dsa $X "/import $I" >> /home/mt/wlists.output

  done

  for I in `find /home/mt/ -name "spwlists?????.rsc" -printf "%f\n" | sort`; do

    echo "delete $I" | ftp $X 21

    rm -f $I;

  done

done

TODO

    When we have more attributes to manipulate from script a STATIC address list it will be reduce the cpu usage a lot.
    If MT adds a counter to the address list, then we can controlling how many times the ip in the smtpok address list is hitting, so, we can assuming that if there are more than 100.000 hits in a particular ip then this server is not spammer and we can move to a white list whit a timeout of 1 month.
    When the dnsbl server returns an ip like 10.0.0.7 or 10.0.0.10 is definitively a BIG SPAMMER. To know this it need to fix the itwbspam script to accept more than one ip from a dnsbl server.

P.S.: if you add modifications in my scripts, please share it whit us !!! :lol:
P.S.: if you add more black lists, please share it whit us !!! :lol:
P.S.: if you add more white lists, please share it whit us !!! :lol:

Enjoy,
Bernardo Pita
Argentina

2 Comments

  • At 2009.12.24 15:35, Andi Sugandi said:

    Pak script ini hanya jalan di 2.9.x.
    Saya testing di mikrotik 3 & 4 script “spamtocheck” dan lainya ada yang tidak jalan, sepertinya ada tag “get” atau “find” tidak cocok di dijalankan MIKROTIK 3&4.
    Apakah ada yang bisa bantu..?

    • At 2013.01.23 07:47, modifyinternal said:

      I was examining some of your posts on this website and I believe this site is real informative ! Continue posting .

      (Required)
      (Required, will not be published)