My experiences from setting up Gnus with nnimap and Courier IMAP

My experiences from setting up Gnus with nnimap and Courier IMAP

A few days before this document was written, I decided to put some time on trying to use a IMAP connection to the IMAP server of Lysator instead of my nnml mail storage in Gnus.

This is my notes from this process. There are some other documents on the subject already out there, but I had to do some creative Googling before getting it to work, so I thought I should assemble some of the solutions that came forward in one document, from my point of view. Hopefully it's of help for somebody.

Why switch to IMAP?

The basic reason I wanted to switch to IMAP is that the storage system of the mailserver is much safer than my disks at home, and I have a lot of mail i don't want to loose because of a disk crash.

Another reason was to be able to read my mail from more locations. With the setup I had before, I had to login to my computer using ssh in order to read my mail in a decent way. Now I'm able to read my mail from various places, regardless of the status of the network to my home computer (it's VERY bad sometimes). Also, we have a Webmail solution that connects to the IMAP server, which means I can read my mail using any computer with a Web browser.

My previous setup

I'm a heavy-duty user of splitting, and my nnmail-split-fancy variable was around 50 entries long. This meant splitting had to work as well when using IMAP, but the documentation about Gnus and IMAP said there was fancy splitting in nnimap as well. We'll talk about how to get that working later...

Basic setup

Since I wanted to keep my existing setup while experimenting in order to be able to go back to it if IMAP was to slow, or didn't work or whatever I started by adding a nnimap select method to my gnus-secondary-select-method. It now looks like this:
(setq gnus-secondary-select-methods '((nnimap "Example"
			              (nnimap-address "imap.example.com")
					      (nnimap-authenticator login)
					      (nnimap-stream ssl)
					      (remove-prefix "INBOX.")
					      (nnimap-authinfo-file
			              "/home/erik/.imap-authinfo"))))

Note the (remove-prefix "INBOX."). That's a special thing I'll talk about later.

Also note the (nnimap-stream ssl). This means we will use an encrypted connection for the data, not transmitting any passwords or mails in cleartext. The important thing here is the password, the mails have already traveled the internet in cleartext. We will do some more ssl customization later, but for now we'll stick with this.

The file /home/erik/.imap-authinfo have the following contents:

machine imap.example.com login <username> password <password>
Where <username> and <password> of course are replaced by the username and password you have on your IMAP server. Ask your systems administration for this information if you don't know.

Splitting

As mentioned before, I'm a splitting-addict :). Here's the guts of my splitting settings in .gnus:

(setq nnimap-split-inbox
        '("INBOX" ))

(setq nnimap-split-predicate "UNDELETED")

(setq nnimap-split-crosspost nil)

(setq nnimap-split-rule '(("Example" ("INBOX" nnimap-split-fancy)))
      nnimap-split-fancy 
      '(| (any "auser@example\\.com" "INBOX.mail")
	  ("X-Spam-Status" "Yes.*" "INBOX.Spamassassined")
	  ("subject" "Backup report" "INBOX.backupreports")
	  ("from" "MAILER-DAEMON@example\\.com" 
	   "INBOX.example.mailer-daemon")
	  (any "myaddr@example.com" "INBOX.mail")
	  "INBOX.mail.unsorted"))
	   

Splitting-related variables explained

The nnimap-split-inbox decides which mailboxes Gnus will try to split mail from. It seems to be wise to keep only the INBOX in this list, and read your personal mail from a box that Gnus split to, not from.

The nnimap-split-predicate variable decides how Gnus searches for mails in the boxes defined in nnimap-split-inbox that should be splitted. This variable has a default value of "UNSEEN UNDELETED" which is IMAP-ish for "Check all mails that neither are read nor marked to be deleted". I find it wiser to set it to "UNDELETED" which means even mail I've read in the box are examined for splitting. This means that if I check my mail with a client that can't do splitting (a web-based client for example) and read some of them, they will still be placed in the correct box when I start my Gnus the next time. Very handy.

The nnimap-split-rule decides what kind of splitting is applied to a certain mailbox on a certain server. At least, this combination of nnimap-split-rule and nnimap-split-inbox works as it should. If I don't define nnimap-split-inbox, it won't work. Strange. Anyway, as you can see, for "Example" which is the name we gave our connection in the select method, the splitting is to be executed via the nnimap-split-fancy function.

Finally, the nnimap-split-fancy variable (that is, there is both a function and a variable with the same name. A bit confusing.) decides how the splitting should occur.

I let all mail marked as SPAM by Spamassassin go into a special box. Before the Spamassassin rule there's a special exception for an address I get personal mail from, but since his mailsystem sucks badly it almost always gets marked as spam by Spamassassin. The rules are executed in the order they occur, so by placing this rule before the Spamassassin rule, it will match the mail and put it in the correct box.

There are also some other rules for various mailtypes I get. Finally, the two last lines split mail for my personal mail address into the INBOX.mail box, and all other mail goes into a special box that serves as a signal that there was no appropriate splitting rule.

Early trouble with splitting

When I tried this setup (well, at least a similar setup) I got the error message Wrong type argument: listp, nnimap-split-fancy. This turned out to be an error in version 5.8.7 of Gnus. I upgraded to v5.8.8 which solved that problem.

Respooling doesn't work!

As a splitting-addict I'm also very fond of the respooling feature of Gnus, most often bound to 'B r'. This is a feature that is very handy when you find that you've missed some case in the splitting rules, and a whole bunch of mail end up in the wrong group, for example in the mail.unsorted group I defined above. The normal way to deal with this is to redefine the splitting-variable to cover the new case, test the splitting code with 'B t' on a mail and then do 'B r' on the mails that reside in the wrong group. This applies the splitting rules again, and moves the mail to the correct group.

This doesn't work with nnimap, by reasons I'm not aware of. Apparently, a lot of the respooling code is nnml-specific.

At first, I thought this was really annoying since I really like that feature. Then Niels Möller pointed out that you can move the mail to the INBOX. That way, Gnus will respool it the next time you fetch mail. Almost as handy, actually.

NAMESPACE

As you might have noticed, all the boxes I split mail to have a name that starts with "INBOX.". This is an effect of the fact that I run this Gnus setup against a Courier IMAP server. Courier has a namespace (which is an IMAP term) that is "INBOX.", basically meaning that all mailboxes have names that start with "INBOX.". A well-written client asks the IMAP-server for the namespace, and presents the mailboxes to the user with names that don't start with "INBOX." since it's a bit ugly.

The nnimap backend doesn't have this functionality, so all mailboxes need to have names the begin with "INBOX.". This is where some clever code by David Kågedal enters the scene:


(setq gnus-group-line-format "%M%S%5y/%-5t: %uG %D\n")
(defun gnus-user-format-function-G (arg)
  (concat (car (cdr gnus-tmp-method)) ":"
          (or (gnus-group-find-parameter gnus-tmp-group 'display-name)
              (let ((prefix (assq 'remove-prefix (cddr gnus-tmp-method))))
                (if (and prefix
                         (string-match (concat "^\\("
                                               (regexp-quote (cadr prefix))
                                               "\\)")
                                       gnus-tmp-qualified-group))
                    (substring gnus-tmp-qualified-group (match-end 1))
                  gnus-tmp-qualified-group)))))
Now, the gnus-group-line-format is just an example. The important thing in it is that we've replaced the normal %G in it with %uG which instructs Gnus to run the function gnus-user-format-function-G to get the string to put in the group-line instead of the string "%uG".

As you might remember from basic setup we defined (remove-prefix "INBOX."). This is used in the function above to remove the part "INBOX." from the presentation of the group name in the group buffer, which makes it easier to read.

Trouble with moving and copying articles

Now my Gnus was happily splitting mail from INBOX to other groups, creating the groups if neccesary. Good. Now I tried to copy a mail from one nnimap group to another. Didn't work. Gnus just hung waiting forever.

Moving articles between nnimap-groups strangely seemed to work as it should, however moving or copying from a nnml group to a nnimap group didn't. Those operations hung forever as well.

Weird.

After some heavy debugging and trussing on the server side I came to the conclusion that the length value Gnus was sending the server when doing APPEND was something like a factor 10 wrong. It was way to large which made Courier wait for more input from Gnus until the timeout for long inactivity from the client was triggered, and Courier closed the session. Now when I knew why, I only needed to know what made Gnus do this.

It turned out the problem was with the ssl connection, which is made using the s_client part of the openssl program. The program used for connecting is defined in the variable imap-ssl-program defined like this:

(defcustom imap-ssl-program '("openssl s_client -ssl3 -connect %s:%p"
			      "openssl s_client -ssl2 -connect %s:%p"
			      "s_client -ssl3 -connect %s:%p"
			      "s_client -ssl2 -connect %s:%p")
I guess earlier versions of openssl didn't say that much to stdout before doing the connection, but my version (0.9.6) was spitting out information about the certificates of the server, and the fact that it couldn't verify them. By some reason Gnus interpretered this in a way that made it give wrong sizes to Courier later in the session. That's what I call a weird bug.

A quick solution to this was to add the commandline option -quiet to openssl s_client, like this:

(setq imap-ssl-program "openssl s_client -quiet -ssl3 -connect %s:%p")
This made the connection work as it should, and I could move and copy articles as I wanted. Good. However, this doesn't protect against Man In The Middle-attacks, since the certificate of the server isn't verified. Anybody with some kind of access to the network path between the client and server could quite easily eavesdrop the traffic. I don't want that to be possible.

After some fiddling with openssl I got it to verify the server, but it didn't have any good way to tell Gnus about it. There's a -verify flag to openssl s_client but it just complains to standard out if it can't verify, something I usually don't see if I run Gnus. It should exit in that situation, something the manual mentions as a bug.

As a solution to this problem I fiddled a bit with stunnel, another SSL software, and found that the following definition of imap-ssl-program worked as I wanted:

(setq imap-ssl-program "/usr/sbin/stunnel -c -A /etc/certs/beca-root-pem.crt -v 2 -S 0 -f -r %s:%p")
In my setup, /etc/certs/beca-root-pem.crt is a public CA certificate from the local CA here at Linköping University. Since the key of my IMAP-server is signed with this, stunnel can verify with some certainty that it's speaking to the correct server, before transmitting my password. Good!