Thunderbird:IMAP RFC 4551 Implementation

See RFC 4551 and bug 436151 for more details

This page describes my plan for implementing this RFC in the mailnews code.

Overview

This extension to the IMAP protocol is primarily intended for use by mobile devices but it will help Thunderbird users as well. It reduces the amount of data the client needs to sync every time a folder is selected. The server keeps track of modification sequence numbers for every change made to a folder. When a folder is selected, the server returns the highest modification sequence number for that folder. The client can then just do an incremental sync, based on the highest modification sequence number it is aware of. If those two numbers are the same, the client doesn't need to do any synchronization at all.

User Experience

This should all be transparent to the user. If their IMAP server supports this extension, selecting large folders should be quite a bit faster, especially over lower bandwidth connections.

We may need to provide a way for the user to turn off this feature, in case their server implementation has bugs in it. My fervent hope is that if this comes to pass, we can get by with a hidden pref.

How Syncing Works Today

When we open a folder, we list all the message UID's and flags in the folder, e.g.,

UID fetch 1:* (FLAGS)

The IMAP protocol code parses the responses and builds up the nsIImapFlagAndUidState object, which it then passes to nsImapMailFolder::UpdateImapMailboxInfo, via the nsIMailboxSpec argument. UpdateImapMailboxInfo then compares the uids and flags it knows about in the nsIMsgDatabase with the uids and flags the server knows about. It removes headers from the db that don't exist any more on the server, updates the flags of messages that it does know about, and tells the nsImapProtocol object about headers it doesn't know about, so the nsImapProtocol code can issue protocol to fetch those headers.

How Syncing Will Work with an RFC 4551 Supporting Server

First, we detect that the server supports RFC 4551, which it indicates by returning CONDSTORE as one of its capabilities. If so, we need to tell the server that we, the client, are interested in HIGHESTMODSEQ for particular imap folders, e.g.,:

SELECT INBOX (CONDSTORE)

As part of its response, the server will say something like:

HIGHESTMODSEQ 150000000

The mod seq # is an unsigned 64 bit integer (as an aside, we don't typically deal with 64 bit unsigned ints, especially in our db code, so we might just persist this as a string).

If we knew about that mod seq #, then we know we don't need to do any synchronization. If our own highest mod seq # is lower, e.g., 140000000, then we just need to know about the things that have changed since our mod seq #, e.g.,

UID FETCH 1:* (FLAGS) (CHANGEDSINCE 140000000)

This will tell us about all the changes that happened since we last used the folder, i.e., messages added, messages deleted, and flags changed.

We will then need to pass an nsIImapFlagAndUidState object which just contains those changes to the nsImapMailFolder, and it's going to need to know how to do a partial update based on just the change set.

The Expunge Problem

It turns out that RFC 4551 doesn't have a way for the client to tell the difference between unchanged and expunged messages. E.g., if I ask for changed messages since my last known mod seq #, it won't tell me about messages that have been deleted and expunged by an other client.

There is an other proposed RFC that fills this gap, QRESYNC RFC 5162. It is not as widely supported as RFC 4551, but we could add support for it as well (though I don't have access to a server that supports it right now).

Or we could do a UID SEARCH ALL to find the list of existing UIDs. This would generate a lot less traffic than our current UID FETCH 1:* FLAGS.

Or we could simply only skip the synchronization step if the mod seq # hasn't changed, i.e., optimize for the single client case. But we can't do even this if the user is using the IMAP delete model because we can't tell if messages have been expunged or not (rfc 4551 doesn't say that issuing an EXPUNGE will change the mod seq #).

What I've decided to do for now is simply do a sanity check on the total number of messages in the folder on the server vs. the total number of headers we have in the db - since we do expunges by default, those numbers should be in sync in normal situations, and if they're not, we do a full sync.

More Implementation Details

We need to track the highest mod seq number we've seen. The imap server will return mod sequence numbers with FETCH responses, e.g., FETCH (UID 4 MODSEQ (12121231000))

and we need to record these and update the db with the largest mod seq number we've seen, so that we know what to use as our CHANGEDSINCE paramemter later on. We'll store this in the db folder info of the db. We probably need to add this as an member of the nsIImapFlagAndUidState, or just remember it in the mailbox spec.