path: root/content/devops/my_mail_setup.rst

My personal Email setup - Notmuch, mbsync, postfix and dovecot
##############################################################

:date: 2017-12-23 16:43 +0530
:slug: email_setup
:author: copyninja
:tags: email, notmuch, mbsync
:summary: Brief description of my email setup.

I've been using personal email setup for quite long and have not documented it
anywhere. Recently when I changed my laptop (a post is pending about it) I got
lost trying to recreate my local mail setup. So this post is a self
documentation so that I don't have to struggle again to get it right.

Server Side
===========

I run my own mail server and I use *postfix* as SMTP server and *Dovecot* for
the IMAP purpose. I'm not going into detail of setting those up as my setup was
mostly done by using scripts created by Jonas for *Redpill* infrastructure. What
redpill is?. (In jonas's own words)

   <jonas> Redpill is a concept - a way to setup Debian hosts to collaborate
   across organisations
   <jonas> I develop the concept, and use it for the first ever Redpill
   network-of-networks redpill.dk, involving my own network (jones.dk), my main
   client's network (homebase.dk), a network in Germany including Skolelinux
   Germany (free-owl.de), and Vasudev's network (copyninja.info)

Along with that I have a dovecot sieve filtering to classify on high level mails
into various folders depending on from where they originate. All the rules live
in the `~/dovecot.sieve` file under every account which has a mail address.

Again I'm not going into detail of how to set these things up, as its not goal
of my this post.

On my Laptop
============

On my laptop I've following 4 parts setup

1. Mail syncing : Done using mbsync command
2. Classification: Done using notmuch
3. Reading: Done using notmuch-emacs
4. Mail sending: Done using postfix running as relay server and SMTP client.

Mail Syncing
============

Mail syncing is done using `mbsync` tool, I was previously user of offlineimap
and recently switched to `mbsync` as I felt it more lighter and simpler to
configure than `offlineimap`. `mbsync` command is provided by package `isync`.

Configuration file is *~/.mbsyncrc*. Below is my sample content with some
private things redacted.

.. code-block:: ini

   IMAPAccount	copyninja
   Host imap.copyninja.info
   User vasudev
   PassCmd	"gpg -q --for-your-eyes-only --no-tty --exit-on-status-write-error --batch --passphrase-file ~/path/to/passphrase.txt -d ~/path/to/mailpass.gpg"
   SSLType IMAPS
   SSLVersion TLSv1.2
   CertificateFile /etc/ssl/certs/ca-certificates.crt


   IMAPAccount gmail-kamathvasudev
   Host imap.gmail.com
   User kamathvasudev@gmail.com
   PassCmd "gpg -q --for-your-eyes-only --no-tty --exit-on-status-write-error --batch --passphrase-file ~/path/to/passphrase.txt -d ~/path/to/mailpass.gpg"
   SSLType IMAPS
   SSLVersion TLSv1.2
   CertificateFile /etc/ssl/certs/ca-certificates.crt

   IMAPStore copyninja-remote
   Account copyninja

   IMAPStore gmail-kamathvasudev-remote
   Account gmail-kamathvasudev

   MaildirStore copyninja-local
   Path ~/Mail/vasudev-copyninja.info/
   Inbox ~/Mail/vasudev-copyninja.info/INBOX

   MaildirStore gmail-kamathvasudev-local
   Path ~/Mail/Gmail-1/
   Inbox ~/Mail/Gmail-1/INBOX

   Channel copyninja
   Master :copyninja-remote:
   Slave :copyninja-local:
   Patterns *
   Create Both
   SyncState *
   Sync All

   Channel gmail-kamathvasudev
   Master :gmail-kamathvasudev-remote:
   Slave :gmail-kamathvasudev-local:
   # Exclude everything under the internal [Gmail] folder, except the interesting folders
   Patterns * ![Gmail]*
   Create Both
   SyncState *
   Sync All

Explanation for some interesting part in above configuration. One is the
*PassCmd* which allows you to provide shell command to obtain the password for
the account. This avoids filling in the password in configuration file. I'm
using symmetric encryption with gpg and storing password some where on my disk.
Which is of course just safe guarded by Unix ACL.

I actually wanted to use my public key to encrypt the file but unlocking the
file when script is run in background or via systemd looks difficult (or looked
nearly impossible). If you have better suggestion I'm all ears :-).

Next instruction part is *Patterns*. This allows you to selectively sync mail
from your mail server. This was really helpful for me to exclude all crappy
*[Gmail]/* folders.


Mail Classification
===================

Once mail is locally on your device, we need a way to read the mails easily in a
mail reader. My original setup was serving synced *Maildir* using local dovecot
instance and read it in *Gnus*. This setup was bit of a over kill with all
server software setups but inability of Gnus to not cope well with *Maildir*
format this was best way to do it. This setup also has a disadvantage, that is
searching a mail quickly when you have huge pile of mail to go through. This is
where *notmuch* comes into picture.

*notmuch* allows me to easily index through Gigabytes of my mail archives and
get what I need very easily. I've created a small script which combines
executing of `mbsync` and `notmuch` execution. I tag mails based on the Maildirs
which are actually created on server side using dovecot sieve. Below is my full
shell script which is doing task of syncing classification and deleting of
spams.

.. code-block:: shell

   #!/bin/sh

   MBSYNC=$(pgrep mbsync)
   NOTMUCH=$(pgrep notmuch)

   if [ -n "$MBSYNC" -o -n "$NOTMUCH" ]; then
      echo "Already running one instance of mail-sync. Exiting..."
	    exit 0
   fi

   echo "Deleting messages tagged as *deleted*"
   notmuch search --format=text0 --output=files tag:deleted |xargs -0 --no-run-if-empty rm -v

   echo "Moving spam to Spam folder"
   notmuch search --format=text0 --output=files tag:Spam and \
     to:vasudev@copyninja.info | \
       xargs -0 -I {} --no-run-if-empty mv -v {} ~/Mail/vasudev-copyninja.info/Spam/cur
   notmuch search --format=text0 --output=files tag:Spam and
     to:vasudev-debian@copyninja.info | \
        xargs -0 -I {} --no-run-if-empty mv -v {} ~/Mail/vasudev-copyninja.info/Spam/cur


   MDIR="vasudev-copyninja.info vasudev-debian Gmail-1"
   mbsync -Va
   notmuch new

   for mdir in $MDIR; do
       echo "Processing $mdir"
       for fdir in $(ls -d /home/vasudev/Mail/$mdir/*); do
         if [ $(basename $fdir) != "INBOX" ]; then
             echo "Tagging for $(basename $fdir)"
             notmuch tag +$(basename $fdir) -inbox -- folder:$mdir/$(basename $fdir)
         fi
       done
   done

So before running mbsync I search for all mails tagged as *deleted* and delete
them from system. Next I look for mails tagged as *Spam* on both my accounts and
move it to Spam folder. Yeah you got it right these are mails escaping the spam
filter and landing in my inbox and personally marked as Spam.

After running mbsync I tag mails based on their folder (searching string
*folder:*). This allows me easily get contents of lets say a mailing list
without remembering the list address.

Reading Mails
=============

Now that we have synced and classified mail its time to setup the reading part.
I use notmuch-emacs interface to read the mails. I use *Spacemacs* flavor of
emacs so I took some time to write down the a private layer which brings
together all my keybindings and classification in one place and does not clutter
my entire *.spacemacs* file. You can find the code for my private layer in
`notmuch-emacs-layer repository
<https://source.copyninja.info/notmuch-emacs-layer.git/>`_

Sending Mails
=============

Well its not sufficient that if we can read mails, we need to be able to reply
to mail. And this was the slightly tricky part where I recently got lost and had
to write this post so that I don't forget it again. (And of course don't have to
refer some outdated posts on web).

My setup to send mails is using *postfix* as SMTP client with my own SMTP server
as relayhost for it. The problem of relaying is it's not for the hosts with
dynamic IP. There are couple of ways to allow hosts with dynamic IP to use relay
servers, one is put the IP address from where mail will originate into
*my_network* or second use SASL authentication.

My preferred way is use of SASL authentication. For this I first had to create a
separate account one for each machine which is going to relay the mails to my
main server. Idea is to not use my primary account for SASL authentication.
(Originally I was using primary account, but Jonas gave this idea of account per
road runner).

.. code-block:: shell

   adduser <hostname>_relay

Here replace <hostname> with name of your laptop/desktop or whatever you are
using. Now we need to adjust postfix to act as relaying server. So add following
lines to postfix configuration

.. code-block:: ini

   # SASL authentication
   smtp_sasl_auth_enable = yes
   smtp_tls_security_level = encrypt
   smtp_sasl_tls_security_options = noanonymous
   relayhost = [smtp.copyninja.info]:submission
   smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd

So here relayhost is the server name which your postfix instance will be using
to relay mails forward into internet. *:submission* part tells postfix to
forward mail on to port 587 (secure). *smtp_sasl_tls_security_options* is set to
disallow anonymous connection. This is must so that relay server trusts your
mobile host and agrees to forward the mail for you.

*/etc/postfix/sasl_passwd* is the file where you need to store password for
account to be used for SASL authentication with server. Put following content
into it.

.. code-block:: conf

   [smtp.example.com]:submission    user:password


Replace smtp.example.com with your SMTP server name which you have put in
*relayhost* configuration. Replace user with <hostname>_relay user you created
and its password.

To secure the sasl_passwd file and create a hash of it for postfix use following
command.

.. code-block:: shell

   chown root:root /etc/postfix/sasl_passwd
   chmod 0600 /etc/postfix/sasl_passwd
   postmap /etc/postfix/sasl_passwd

The last command will create /etc/postfix/sasl_passwd.db file which is hash of
your file /etc/postfix/sasl_passwd with same owner and permission. Now reload
the postfix and check if mail makes out of your system using mail command.

Bonus Part
==========

Well since I've a script created above bringing together mail syncing and
classification. I went ahead and created a systemd timer to periodically sync
mails in the background. In my case every 10 minutes. Below is *mailsync.timer*
file.

.. code-block:: ini

   [Unit]
   Description=Check Mail Every 10 minutes
   RefuseManualStart=no
   RefuseManualStop=no

   [Timer]
   Persistent=false
   OnBootSec=5min
   OnUnitActiveSec=10min
   Unit=mailsync.service

   [Install]
   WantedBy=default.target

Below is *mailsync.service* which is needed by mailsync.timer to execute our
scripts.

.. code-block:: ini

   [Unit]
   Description=Check Mail
   RefuseManualStart=no
   RefuseManualStop=yes

   [Service]
   Type=oneshot
   ExecStart=/usr/local/bin/mail-sync
   StandardOutput=syslog
   StandardError=syslog

Put these files under /etc/systemd/user and run below command to enable them.

.. code-block:: shell

   systemctl enable --user mailsync.timer
   systemctl enable --user mailsync.service
   systemctl start --user mailsync.timer

So that's how I've sync and send mail from my system. I came to know about
*afew* from Jonas Smedegaard who also proof read this post. So next step I will
try to improve my notmuch configuration using afew and of course a post will
follow after that :-).