Installing your own Linux mail server

Recently, a friend of mine asked me for help with installing his own mail server. Since this is can be a long complicated process that involves many different components to set up properly, and since I don't perform this frequently enough to remember all the steps, I decided to write it down for future reference.

As I am a Gentoo user, these instructions are written for Linux Gentoo. In general, this will mean that the instructions will use emerge rather than apt-get or any of the other common package managers. However, in the cases where the instructions end up being Gentoo-specific, I will also make sure to describe how to perform the same steps on Debian-based distributions, as Debian is another distribution I commonly have to use.

As of writing, we will be installing the following:

  • Postfix 3.2.4
  • Dovecot 2.2.32
  • Amavisd-new 2.11.0-r3
  • ClamAV 0.99.2-r1
  • SpamAssassin 3.4.1-r19
  • Pyzor 1.0.0
  • Razor 2.85-r2
  • OpenDKIM 2.10.3-r3
  • Certbot 0.19.0

You should have reverse DNS set up for the domain(s) you are going to use. Furthermore, you should have the following installed:

  • PostgreSQL 9.6.4
  • nginx 1.12.2 (or Apache 2.4.27-r1)

Let's get started.

Mail storage

Instead of creating a separate UNIX account for each user, we will set up a single UNIX user called vmail that will own all the virtual mailboxes on our system. We will assign 5000 as a unique UID and GID to this user, assuming that it is still freely available. Since this UNIX account is not supposed to be used, we will set its shell to /bin/false.

groupadd -g 5000 vmail
useradd -m -d /var/vmail -s /bin/false -u 5000 -g vmail vmail

Now we have to set up the storage space that will contain our mailboxes. This could just be a plain directory, but also a NFS share or a partition on a hard drive. After we have decided the storage to use, we have to set up the ownership and permissions appropriately.

chown vmail: /var/vmail
chmod 2770 /var/vmail

Postfixadmin

While we could manage domains, e-mail addresses, etc. in plain text files, it makes more sense to store them in a database such as PostgreSQL. To simplify the installation, we will be using postfixadmin which is a web interface that allows us to manage these in a straight-forward manner. Another benefit of using postfixadmin is that we can use it to set up the tables in our database using a well-known format. First, we will install postfixadmin as follows:

emerge -a postfixadmin

We will use webapp-config to install postfixadmin. Since this is going to be a management interface that we will use to manage e-mail addresses, domains, etc., we will only make it available to localhost. We will use ssh to forward the port later on such that we can access it externally without compromising the security.

webapp-config -h localhost -d postfixadmin -I postfixadmin 2.3.8

Next we will set up the database as well as a database user for postfixadmin to use for administration and a database user postfix to only access the database:

create -U postgres --pwprompt postfixadmin
create -U postgres --pwprompt postfix
createdb -U postgres --owner=postfixadmin postfix

The postfix user also needs some permission to access the tables:

psql -U postfixadmin postfix
GRANT SELECT ON alias TO postfix;
GRANT SELECT ON domain TO postfix;
GRANT SELECT ON mailbox TO postfix;
GRANT SELECT on alias_domain TO postfix;

Now that we have set up the database together with the database users, we have to configure /var/www/localhost/htdocs/postfixadmin/config.inc.php or /etc/postfixadmin/config.inc.php, depending on where it is installed, to use the database. In order to use hash algorithms compatible with Dovecot, we configure postfixadmin to use doveadm as follows:

$CONF['encrypt'] = 'dovecot:CRYPT';
$CONF['dovecotpw'] = "/usr/bin/doveadm pw";

To list the supported hash algorithms, run the following command:

doveadm pw -l

Once you are done configuring postfixadmin, set $CONF['configured'] = false; to true.

After the database has been set up, we can forward port 80 to 8080 locally using ssh to access the web interface:

ssh -L 8080:localhost:80 hostname

Visit http://localhost:8080/postfixadmin/setup.php in your web browser. If everything is all right,you should see a list of 'OK' messages now. When you visit the page for the first time, it will attempt to create the database structure. Enter a password to use for the setup and submit the form. This will simply generate a hash that you have to save in the config file. After adding the hash to the config file, you can now use the setup password to create a superadmin account. Visit http://localhost:8080/postfixadmin/login.php and log in with the account you have just created. Now you will be presented with an interface that will allow you to manage your domains, e-mail addresses, etc.

Debian:

On Debian, postfixadmin can be configured while installing it as follows:

apt-get install postfixadmin

This will prompt a wizard that will help you set up the database. In case of a mistake, you can always run the following command to reconfigure it:

dpkg-reconfigure postfixadmin

Postfix

Now that we have set up the database, we will install our SMTP server. We will be using Postfix as our SMTP server of choice. Install it as follows:

echo "mail-mta/postfix dovecot-sasl" >> /etc/portage/package.use
emerge -a postfix

After Postfix has been installed, we have to tell it to use our vmail UNIX user and where e-mails should be stored. Add the following to /etc/postfix/main.cf:

virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
virtual_mailbox_base = /var/vmail
virtual_transport = virtual

Next we will have to write the PostgreSQL queries such that Postfix can query the database.

mkdir /etc/postfix/pgsql
cat >/etc/postfix/pgsql/virtual_mailbox_domains.cf <<EOF
user = postfix
password = ${postfix_password}
dbname = postfix
query = SELECT domain FROM domain WHERE domain = '%s' AND backupmx = '0' AND active = '1';
EOF
cat >/etc/postfix/pgsql/virtual_mailbox_maps.cf <<EOF
user = postfix
password = ${postfix_password}
dbname = postfix
query = SELECT maildir FROM mailbox WHERE local_part = '%u' AND domain = '%d' and ACTIVE = '1';
EOF
cat >/etc/postfix/pgsql/virtual_alias_maps.cf <<EOF
user = postfix
password = ${postfix_password}
dbname = postfix
query = SELECT goto FROM alias WHERE address = '%s' AND active = '1';
EOF
cat >/etc/postfix/pgsql/virtual_domain_alias_maps.cf <<EOF
user = postfix
password = ${postfix_password}
dbname = postfix
query           = SELECT CONCAT('%u', '@', target_domain) FROM alias_domain WHERE alias_domain = '%d' AND active = '1';
EOF
chgrp -hR postfix /etc/postfix/pgsql

And then we edit /etc/postfix/main.cf to tell Postfix to use these queries:

virtual_mailbox_domains = pgsql:/etc/postfix/pgsql/virtual_mailbox_domains.cf
virtual_mailbox_maps = pgsql:/etc/postfix/pgsql/virtual_mailbox_maps.cf
virtual_alias_maps =
        pgsql:/etc/postfix/pgsql/virtual_alias_maps.cf,
        pgsql:/etc/postfix/pgsql/virtual_domain_alias_maps.cf

In the same file edit the following to use your domain name, but keep the rest set to their defaults:

myhostname = example.com
myorigin = example.com
relayhost =
mynetworks = 127.0.0.0/8
mydestination =
inet_protocols = ipv4 ipv6

alias_maps = hash:/etc/postfix/aliases
alias_database = $alias_maps

Finally, we do not want to fall prey to users with malicious intent. As many attackers will use fairly simple approaches to try and abuse our server, we will raise the bar by checking for some basic conditions that they will commonly fail to meet and use SASL to implement SMTP authentication. In addition, we will also use zen.spamhaus.org and bl.spamcop.net as blacklists to reject well-known spammers.

disable_vrfy_command = yes
smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_helo_restrictions =
        permit_mynetworks,
        reject_non_fqdn_helo_hostname,
        reject_invalid_helo_hostname,
        permit
smtpd_recipient_restrictions =
        permit_sasl_authenticated,
        reject_unauth_pipelining,
        reject_invalid_hostname,
        reject_non_fqdn_hostname,
        reject_non_fqdn_recipient,
        reject_unknown_recipient_domain,
        reject_unauth_destination,
        permit_mynetworks,
        reject_rbl_client zen.spamhaus.org,
        reject_rbl_client bl.spamcop.net,
        permit

Unlike the more complicated Cyrus SASL, we will use Dovecot's SASL in Postfix to set up SMTP authentication:

broken_sasl_auth_clients = no
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain =
smtpd_sasl_authenticated_header = no

Make sure smtp is enabled in /etc/postfix/master.cf:

smtp      inet  n       -       -       -       -       smtpd

Dovecot

We will be installing and configuring our IMAP and POP3 server to be able to log in onto our accounts and read our e-mails. First install Dovecot:

emerge -a dovecot

First, we make sure that Dovecot knows where the e-mails are stored. Add the following to /etc/dovecot/conf.d/10-mail.conf:

mail_location = maildir:/var/vmail/%u
mail_privileged_group = vmail
mail_uid = vmail
mail_gid = vmail
first_valid_uid = 5000
last_valid_uid = 5000

We will also set up a private inbox for our users:

namespace inbox {
    type private
    separator = /
    prefix =
    inbox = yes
    hidden = no

    mailbox Trash {
        auto = no
        special_use = \Trash
    }

    mailbox Drafts {
        auto = no
        special_use = \Drafts
    }

    mailbox Sent {
        auto = subscribe # autocreate and autosubscribe the Sent mailbox
        special_use = \Sent
    }

    mailbox "Sent Messages" {
        auto = no
        special_use = \Sent
    }

    mailbox Spam {
        auto = create # autocreate Spam, but don't autosubscribe
        special_use = \Junk
    }

    mailbox virtual/All { # if you have a virtual "All messages" mailbox 
        auto = no
        special_use = \All
    }
}

Next, we configure SASL in /etc/dovecot/conf.d/10-auth.conf. We will set the auth_mechanisms to both plain and login:

auth_mechanisms = plain login

Since we will also be using the PostgreSQL database, make sure to include auth-sql.conf.ext instead of auth-system.conf.ext:

#!include auth-system.conf.ext
!include dovecot-sql.conf.ext

In /etc/dovecot/auth-sql.conf.ext we will configure both the queries to use as well as the credentals to connect to the database:

driver = pgsql
connect = host=localhost dbname=postfix user=postfix password=${postfix_password}
default_pass_scheme = CRYPT
password_query = SELECT password FROM mailbox WHERE username='%u' AND active='1'

In /etc/dovecot/conf.d/auth-sql.conf.ext make sure to use sql for the passdb entry, but static for the userdb as follows:

passdb {
    driver = sql
    args = /etc/dovecot/dovecot-sql.conf.ext
}

userdb {
    driver = static
    args = uid=vmail gid=vmail home=/var/vmail/%u
}

Finally, in /etc/dovecot/conf.d/10-master.conf, enable the following services:

service lmtp {
    unix_listener lmtp {
        mode = 0666
        user = postfix
        group = postfix
    }
}

service auth {
    unix_listener auth-userdb {
        mode = 0666
        user = vmail
    }

    unix_listener /var/spool/postfix/private/auth {
        mode = 0666
        user = postfix
        group = postfix
    }

    user = dovecot
}

service auth-worker {
    user = vmail
}

Now you should be able to start both Postfix and Dovecot:

/etc/init.d/postfix start
/etc/init.d/dovecot start

Debian:

On Debian, you can simply install Dovecot as follows:

apt-get install dovecot-imapd dovecot-pop3d

To start Dovecot you can use dovecot, and doveadm reload and doveadm stop to reload the configuration or stop Dovecot respectively.

Amavisd, Clamav and Spamassassin

Next we will add virus scanning and spam filtering to our mail server. The main packages that we will have to install are Amavisd, ClamAV, SpamAssassin. For ClamAV and SpamAssassin, we will simply enable the USE-flags. If you also want support for OpenDKIM, then also enable the dkim USE-flag. We enable this USE-flag up front, to prevent a re-installation of Amavisd. To further improve spam filtering we will also install Razor and Pyzor:

echo "mail-filter/amavisd-new clamav dkim spamassassin" >> /etc/portage/package.use
emerge -a amavisd-new pyzor razor

As Amavisd uses the SpamAssassin libraries, we don't have to start or configure SpamAssassin. To enable Razor and Pyzor, run the following commands:

su - amavis -s /bin/bash
razor-admin -create
razor-admin -register
pyzor discover

As the defaults for ClamAV will be fine for our needs, we only have to make sure that the UNIX accounts are in the same group such that they have access to the files:

adduser clamav amavis
adduser amavis clamav

Then we start up ClamAV:

/etc/init.d/clamd start

In /etc/amavis/conf.d/15-content_filter_mode make sure to uncomment the following two lines to enable virus scanning and spam filtering:

# Default antivirus checking mode
# Uncomment the two lines below to enable it
#

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);


#
# Default SPAM checking mode
# Uncomment the two lines below to enable it
#

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

Then start Amavisd:

/etc/init.d/amavisd-new start

Add the following immediately below pickup in /etc/postfix/master.cf to make sure we don't scan our own generated messages:

  -o content_filter=
  -o receive_override_options=no_header_body_checks

Finally, we have to tell Postfix to route e-mails via Amavisd for scanning. Append the following lines to the end of '/etc/postfix/master.cf`:

smtp-amavis     unix    -       -       -       -       2       smtp
  -o smtp_data_done_timeout=1200
  -o smtp_send_xforward_command=yes
  -o disable_dns_lookups=yes
  -o max_use=20

127.0.0.1:10025 inet    n       -       -       -       -       smtpd
  -o content_filter=
  -o local_recipient_maps=
  -o relay_recipient_maps=
  -o smtpd_restriction_classes=
  -o smtpd_delay_reject=no
  -o smtpd_client_restrictions=permit_mynetworks,reject
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o smtpd_data_restrictions=reject_unauth_pipelining
  -o smtpd_end_of_data_restrictions=
  -o mynetworks=127.0.0.0/8
  -o smtpd_error_sleep_time=0
  -o smtpd_soft_error_limit=1001
  -o smtpd_hard_error_limit=1000
  -o smtpd_client_connection_count_limit=0
  -o smtpd_client_connection_rate_limit=0
  -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

Finally, reload the configuration for Postfix:

/etc/init.d/postfix reload

Debian

On Debian, you can simply install Amavisd, ClamAV, SpamAssassin, Pyzor and Razor as follows:

apt-get install amavisd-new spamassassin clamav-daemon libnet-dns-perl libmail-spf-perl pyzor razor

To also scan various archive formats, it is recommended to install the following:

apt-get arj bzip2 cabextract cpio file gzip nomarch pax rar unrar unzip zip zoo

On Debian, the services are named /etc/init.d/amavis and /etc/init.d/clamav-daemon instead.

SSL

To make the internet a better place, we also want to encrypt our SMTP, IMAP and POP3 connections using a SSL certificate. We'll be using Let's Encrypt free service for this. First, we will install their tools to automatically obtain SSL certificates:

emerge -a certbot certbot-nginx

Then we can invoke certbot with the domains and subdomains we want to obtain a SSL certificate for:

certbot --nginx -d example.com -d www.example.com

The renewal process should be automatic since certbot either uses the systemd timers or a cronjob to check if the SSL certificates have to be renewed every two days. To check if the new renewal process works we can perform a dry run:

certbot renew --dry-run

After we have obtained the SSL certificate, we have to configure Postfix and Dovecot to use it. First, add the following lines to /etc/postfix/main.cf to let it know where to find the keys:

smtpd_use_tls = yes
smtpd_tls_key_file = /etc/letsencrypt/live/example.com/privkey.pem
smtpd_tls_cert_file = /etc/letsencrypt/live/example.com/fullchain.pem
smtpd_tls_security_level=may
smtpd_tls_auth_only = no
smtpd_tls_loglevel = 3
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom

Next, we will enable both submission and smpts in /etc/postfix/main.cf:

submission inet n       -       n       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
smtps     inet  n       -       n       -       -       smtpd
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes

In /etc/dovecot/conf.d/10-ssl.conf, we want to add the following lines to enable SSL and to let Dovecot know where to find our keys:

ssl = yes
ssl_cert = </etc/letsencrypt/live/example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/example.com/privkey.pem

Now that everything has been set up to use SSL, we simply have to reload the configurations for both Postfix and Dovecot:

/etc/init.d/postfix reload
/etc/init.d/dovecot reload

Debian:

On Debian systems, you might have to add a PPA to get certbot:

add-apt-repository ppa:certbot/certbot
apt-get update

Then you can install certbot from the new repository:

apt-get install python-certbot-nginx

DNS

In order for other SMTP servers to be able to send us e-mails, they have to be able to figure out that we are running a SMTP server ourselves. First, add the following MX record to the DNS settings for your domain name:

10 mail.example.com

Modern SMTP servers will also check whether the SMTP server is allowed to send e-mails from the domain it claims to be sending e-mails from. Using the Sender Policy Framework (SPF), we can add a TXT record to the DNS settings of our domain name that tells other SMTP servers that we are allowed to send e-mails:

v=spf1 a mx mx:example.com ~all

OpenDKIM

DomainKeys Identified Mail (DKIM) allows a mail transfer agent (MTA) to verify that the message actually comes from the domain that it claims to come from. This works by setting up a public and private key pair where the public key is available as a DNS record and where the private key is used to sign sent e-mails. The receiving MTA can then perform a DNS look up to get the public key and use this to verify the origin of the e-mail. This not only ensures that spammers can no longer forge headers, but that e-mails can no longer be tampered with. We assume amavisd-new has already been installed with the dkim USE-flag enabled.

First, we have to generate a public and private key pair for our domain:

opendkim-genkey -t -s mail -d example.com

Move the private key to its appropriate place, such that OpenDKIM can use it:

mv mail.private /etc/mail/dkim.key

And add the DNS record described in mail.txt to your domain(s). Then add the following to /etc/postfix/main.cf to link Postfix with OpenDKIM:

milter_protocol = 2
milter_default_action = accept
smtpd_milters = unix:/var/run/opendkim/opendkim.sock
non_smptd_milters = unix:/var/run/opendkim/opendkim.sock

Finally, we can start OpenDKIM and reload Postfix:

/etc/init.d/opendkim start
/etc/init.d/postfix reload

Debian:

On Debian, you simply have to install opendkim as follows:

apt-get install opendkim

Finishing touches

Now that everything has been set up, we want to make sure that the services start up automatically in case of a server reboot:

rc-update add {postfix,dovecot,clamd,amavisd-new,opendkim} default

Congratulations, you now should have a fully functional mail server stack up and running.

Debian:

On Debian systems, installing the packages will automatically add the services to systemd to start up during boot.