MailNews:Better Faster IMAP Plan

From MozillaWiki
Jump to: navigation, search

Leverage Offline capabilities to make online more responsive

The basic idea is to use the offline capabilities of Thunderbird while online to make the UI more responsive, when operating on IMAP messages, out of the box, without the user having to tweak any settings. (Note: this applies to SMTP sending as well)

UX Decisions to make

With an assumption that all operations are to be perceived as more responsive, many of our decisions are made for us. A proxy increases speed also increase the need for good feedback because operations appear to have completed quickly and yet may not have actually taken place.

  • are all operations to be available offline?
  • how do we transition existing users to the new model without causing pain?
  • The offline model will make "Unsent Items" more important -- should we promote it to a high-visibility UI element (like Mail.app's Outbox)?
    • Until we have a failure in sending the Outbox doesn't offer much value, however once a failure occurs it needs to be high-visibility (clarkbw)
    • Do you suggest a virtual Outbox that will show up only there are pending messages? (emre)
  • What UI do we present to allow users to override the auto-download of all message bodies? Is it per folder? Per account? Global?
    • This problem needs to be broken out into what I believe are 2 separate concerns. (clarkbw)
      • Concern 1: Change. Some people will be concerned about this change no matter what and will want an "off switch" or will complain loudly.
      • Concern 2: Space. Other are probably concerned with space issues of auto-downloading, this requires some thought and is most likely the real underlying concern of #1.
      • Concern 3: Policy. Some people may be under a policy that does not allow them to keep a local store of messages. (jaym)
  • Do we give the user feedback when the delete actually happens? (davida: IMO no, only when delete fails)
    • We need to give feedback for every operation but on different levels (clarkbw)
      • Notice feedback for operations that are completed
      • Error feedback (and rollback) for operations that fail
  • Do we make the Trash an offline folder like all other folders? (davida: Why not?)
    • We need to make Trash fast, responsive, and work offline (clarkbw)
  • Are there other imap operations that we would want to do "offline", like renaming a folder, or should we stick with the most often used commands, like reading a message, move, delete, SMTP send, etc?
    • Why not? (clarkbw)
    • Draft Messages


Account Settings The goal of the account settings is to allow people to override the global offline preference on an account by account basis. 

+--------------------------------------------------------------------+
|  Syncing & Disk Space                                              |
+--------------------------------------------------------------------+
|  Message Synchronization                                           |
|    [x] Keep all messages for this account on this computer         |
|                                                   ( Advanced... )  |
|                                                                    |
+--------------------------------------------------------------------+
  • Checkbox (defaults to global offline option)
    • When checked, changes all folders in the account to be sync'd
    • When unchecked changes all folders in account to not be sync'd
  • ( Advanced... )
    • Opens folder selector for changing individual folder setup

Development Strategy

  1. Consider implementing what follows using a "proxy" to mediate between the UI and the IMAP protocol layer (to be reusable w/ other protocols)
  2. Switch pref defaults
    • switch the default for imap folders so that they're configured for offline use out of the box. (code URL?)
    • switch mail.server.default.autosync_offline_stores to true, by default
  3. Do UI operations in offline mode always
    • change imap move/deletes though the UI so that they're done offline, and the offline operation is played back after the next message is displayed, or immediately if there's no next message to display.
  4. Come up with a message header and message body download strategy
    • We probably want to download recent information before older information. The most succinct form of the command to fetch multiple messages (bodies or headers) fetches them in UID, roughly chronological order, using ranges, e.g., 1-20. But you certainly could fetch 20,19,18,17 etc. Are you talking about the new profile case, or the every day case? For the new profile case, a simple thing to do would be to fetch the bodies of the unread messages first, then the bodies of the read messages. Or we could fetch ranges of messages, but start at the end, not the top, e.g., 80-100, 60-79, etc.
    • We should be careful about downloading automatically large message bodies, as they may block IMAP calls.
    • This would allow us to consider applying the same sort of logic to other protocols (NNTP?)
  5. Come up with an offline operation playback strategy
    • May be worth implementing a few and seeing which ones work out best in practice.
  6. Address consequences of preference changes
    • address growth of offline store folder - come up with a compacting strategy that is user-efficient. (once a week? once a month? Once a file reaches some % growth since the last compacting?)
    • Make sure that offline operations that fail give the user appropriate context as to the failure (offer to open up original message when sends fail, etc.)
  7. NB: At this point, the system should be releasable
  8. Consider further enhancements
    • Use the new platform code to detect whether there is a net connection or not, and 1) display that status to the user somehow, and 2) use the status to determine when to replay offline operations.
    • Use the Idle service to download message bodies when the user is idle, or to compact offline stores
    • Breakup the download of message bodies into multiple passes, which would allow the user to sneak in and start reading messages before we've synced the whole folder.
    • Playback offline operations when the user is idle, though I think we'd run into less issues if we tried to do the offline playback closer to the UI event.

Emre: Proposal for Implementation

I like to break down these requirements into 3 distinct implementation tasks:

  • Offline operation playback feature implementation (needs a better name)
  • Preemptive/Automatic message download feature implementation
  • Background message send feature implementation
  1. Offline operation playback feature Delete operation covers two different operations at the same time: Delete and Move. These two server-side operations cover the majority of use cases that require better offline support. Since it provides good granularity for deployment and testing, it makes sense to start implementing "Proxy" mechanism focusing on Delete operation first. Possible milestones are;
    1. Implement offline support for "Delete" operation:
      • Deleting messages, considering two flavors of delete;
        • Move to Trash
        • Delete immediately
      • Deleting folders
    2. Implement UI feedback mechanism (UI elements and logic)
      • consider strategies for 3 different type of errors
        • immediate errors returned by imapService->PlaybackAllOfflineOperations method. Mostly standard NSPR errors such as out of memory etc..
        • network errors that will be returned by OnStopRunningUrl callback in exit code parameter
        • server errors that will be returned by OnStopRunningUrl callback in exit code parameter
      • Modify error notification mechanism in imap protocol layer
    3. Implement offline support for other operations (rename, flags, copy, folder creation)
  2. Preemptive/Automatic message download feature Possible milestones are:
    1. Modify current offline features
      • Switch the default for imap folders so that they're configured for offline use out of the box
      • Switch mail.server.default.autosync_offline_stores to true, by default
    2. Implement automatic compacting mechanism for mbox files
    3. Re-factor current offline mechanism to implement strategy-based, automatic message body/attachment download. Couple ideas are:
      • Prioritize by date
      • Prioritize by sender
      • Prioritize by size, or attachment count
      • Idle service usage
      • Multi pass download for messages with multiple attachments
    4. Implement UI elements and logic
  3. Background message send feature Possible milestones are:
    1. Implement a "Unsent" folder UI element for IMAP folders
    2. Modify existing code to:
      • Create a new nsMsgFolder type
      • Show pending messages inside this folder - allow limited user interaction (cancel only?)
      • Don't show "sending" dialog anymore
  • Shortcomings of the current implementation
    • Sends messages one by one
    • Can't send just a single message from unsent folder
    • Doesn't allow to edit a message in unsent folder

1) Offline Operation Playback

[Emre May 21st, 2008] bug 435153 has been filed.

[Emre May 22nd, 2008] This task list is now obsolete.

  1. Implement Proxy mechanism (Estimation: ?)
    • Shall map local folder model to server folder model, vice versa.
    • Shall execute commands on background - decouple folder event from protocol command.
    • Shall notify UI if the operation is not successful.
    • (?) Shall store commands persistently in case that TB exits while there are pending operations in the Q.
    • Shall deal with unsolicited UI events and IMAP server messages.
    • Shall rollback the UI operation if the command is not successful.
    • Shall be smart enough to deal with event chaining, for example; Assume that Delete mode is "MoveToTrash", the user deletes 100 messages. TB moves the messages into the trash immediately and starts the operation on the background. Then the user moves 50th deleted message back into the INBOX and mark it as "Not Read" before the "move to trash" command is sent to the server. Proxy should be smart enough not to lose the message identifier, and change the command chain if necessary - moving a message into trash might change its identifier on the server, so the content of the pending command.
  2. Refactor Imap protocol layer to handle error cases gracefully, such as no-connection, server error, command execution error. Note: Put current threading model into consideration. When the user exceeds the number of cached-connections, all pending operations waiting for this session should be executed before recycling the thread. Thread recycling causes disconnection form the server which might cause an implicit EXPUNGE command on the server. (Estimation: ?)
  3. Implement UI feedback mechanism (Estimation: ?)

Total:

Risks:

Decisions to make

[bienvenu] My suggestion would be to leverage the existing offline support. When the user is using TB in offline mode, we remember all the offline ops in the .msf file, and play them back when online. This gives you persistance, playback, etc.

[Emre May 22nd, 2008] I am convinced that the mechanism we need for this feature has already been implemented in the existing offline support. I am going to leverage it with bienvenu's help.

1) Should we warn the user about pending offline operations before exit?

  • if yes;
    • should we give the user an option to store them locally for the next time?
    • or should we do it automatically
  • if no;
    • What's the right action to take?

Even this is a rare case, it is inevitable: imagine a scenario where the user deletes 200 messages and tries to exit immediately.

[Emre]We should warn the user about pending operations and give the option to either execute them before exit TB, or save them for the next time.

[Emre May 22nd, 2008] Existing offline mechanism stores all the operations persistently in the database and playback them when TB becomes online, or in case of exit, at the startup in UpdateFolder(). AFAIK it doesn't give any kind of warning for pending operations before exit, it does silently. We are going to use the same workflow.

2) Offline operation storage:

  • Is offline operation persistency required?
    • If yes, for which operations? Delete? Rename? Flag change? Tagging?
  • How long should we store it?
  • How to deal with aged pending offline operations?
  • What kind of store we should use? Mork, Sqlite, text file
  • Is security a concern - encryption?

[Emre May 22nd, 2008] No change will be done to the existing offline support in this regard.

3) UI feedback for:

  • Operations that are completed
  • Error feedback (and rollback) for operations that fail
  • Feedback for partially or not downloaded messages (?)

[Emre May 22nd, 2008] No change will be done to the existing behavior in this regard. Error messages will be close to the user actions to prevent confusion. If the operation is failed, a message dialog will be shown (existing behavior) and the operation will be saved to be play-backed until it is successful.

4) When do we playback offline operations: [Emre] We can deal with this problem by making a design that is flexible enough to implement different strategies. Don't believe it would cost us much in term of time.

[Emre May 22nd, 2008] Two possible strategies are;

  • Playback after the next messages has been fetched. Rationale: There is only one session to the server, and commands are sent one by one. Fetching the next message body before playing-back the pending operation decreases wait time for the user.
  • Playback using an one-shot preset timer. Rationale: This is very similar to the first one, but it is not bound to any condition. Existing workflow is to moving/copying/deleting messages on the server then fetching the next message in the list to show. Firing a timer, lets say with a 500ms delay, to playback pending operation gives enough time to UI to fetch the next message (or to execute whatever next step is in the workflow) before the pending op is sent to the server - as I understand it, since all UI events are queued and executed by UI thread, the order of the events is more important than the delay time.

Note that in both strategies, error cases should be handled properly. For example, if the playback request is failed for some reason, TB should give an error message only the first time (clarkbw?), and should suppress the error messages for the consecutive tries.

Design Proposal

See current design [TBD]

2) Preemptive/Automatic Message Download

[Emre May 30th, 2008] bug 436615 has been filed. Marked FIXED 2008-09-17 17:01:35 PDT.

Existing Behavior

Decision table for IMAP message operations as of 6th June 2008:

Copying/Moving an IMAP message
1
2
3
4
5
6
7
8
9
10
11
12
Source folder is in online mode
Y
Y
Y
Y
N
N
N
N
N
N
N
N
Destination folder is in online mode
Y
Y
N
N
Y
Y
Y
Y
N
N
N
N
TB is in online state
Y
N
Y
N
Y
Y
N
N
Y
Y
N
N
Message is available locally - - - -
Y
N
Y
N
Y
N
Y
N
Actions                        
Remove the header from the source folder's database
X2
X2
X2
X2
X2
X2
X2
X2
X2
X2
X2
X2
Add the new header into the destination folder's database
X1
X
X1
X
X1
X1
X
X
X1
X1
X
X
Copy the message into the destination folder's mbox  
X
          
X
    
X
  
  • DELETE and MOVE are the same operations in case that DELETE means "move to Trash"
  • 1 If and only if the destination folder is already selected (having connection to the server)
  • 2 Do not remove if the operation is COPY
Selecting an IMAP message
1
2
3
4
Message folder is in online mode
Y
Y
N
N
TB is in online state
Y
N
Y
N
Actions        
Fetch message from the server
X
-
X
-
Get local copy if available  
X
X
X
Store locally in the folder's mbox    
X1
  
  • 1 Depending on preferences message is not stored locally if non-inline MIME parts (e.g. attachments) size > mime_parts_on_demand_threshold (default ~30KB) OR total message size > limit_offline_message_size (default off). See Advanced and Offline & Diskspace preferences.


Tagging an IMAP message
1
2
3
4
Message folder is in online mode
Y
Y
N
N
TB is in online state
Y
N
Y
N
Actions        
Store the flag on
server
local db
server
local db


Selecting an IMAP folder
1
2
3
4
Message folder is in online mode
Y
Y
N
N
TB is in online state
Y
N
Y
N
Actions        
Fetch message headers from the server
X
  
X
  
Remove headers from the database that do not exist on the server
X
  
X
  
Add new headers to the local database
X
  
X
  
Fetch headers from the local database  
X
  
X
Playback offline operations
X1
  
X1
  

1 If and only if the selected folder is the source folder of the pending offline operation

How it works

Source and Destination folders are offline, autosync_offline_stores is set to true by default, pseudo offline is enabled:

  • When a folder is selected the first time, TB fetches all message keys (a.k.a Messages unique identifier values) belong to this folder from the server (using <tag> UID fetch 1:* (FLAGS) imap command). It keeps the mutual keys intact, adds new keys into the local database, and removes the non-existing ones. Once this operation is completed, it fetches the headers and message bodies belong to the new keys (I think all headers first, bodies later, not sure). Finally, it stores the next UID value to determine whether any messages have been delivered to the mailbox next time it is selected.
  • If another folder is selected before all message bodies are downloaded, it stops downloading, switches to the other folder and starts the same process on this newly selected folder. Once this is completed, it resumes the download of the message bodies belong to the previously selected folder. A progress bar appears at the bottom right side of the window, but it doesn't show any progress.
  • Unless it is in offline state, TB always fetches message bodies from the server even when they are available locally.
  • When messages are moved from a folder to another, TB moves them immediately (Pseudo-Offline) and playback the operation later. Since TB creates fake msg keys during offline move, when the destination folder is selected, all its content are fetched again because of the reason explained above. mbox file for this folder increases as well.
  • Delete is similar to Move operation if delete mode is move-to-trash.

Requirements

  1. TB shall give two options to the user about how to download messages when a new account is created. Options are:
    1. TB downloads all message headers and message bodies. The user has to wait until the downloads are completed.
    2. TB downloads all message headers and starts downloading message bodies on background according to a predefined strategy. if the user selects a message on the list, TB should interrupt the current download without loosing the parts already downloaded, and should start downloading the selected message.
  2. When new messages arrive, TB should download message headers immediately and should start downloading the bodies if the download queue is empty. If the download queue is busy, the decision can be made based on the active strategy (see possible strategies above)
  3. Highest priority should be given always to the selected message.
  4. All operations conducted on IMAP folders should be completed immediately, except copying/moving IMAP messages/folders to another server or account type.
  5. All messages should be stored locally for offline and fast access.
  6. Message header synchronization with the IMAP server shall be done silently, on background.
  7. A new error notification mechanism is needed to accommodate new UI elements.
Message Download Strategy

NOT COMPLETED

 
1
2
3
4
Message is selected by the user
Y
Y
N
N
Sender is in the address book
Y
N
Y
N
Message date
Today
Older
Y
N
Message has attachments
-
-
-
-
Message size is ..
-
-
-
-
MIME disposition tag ..
-
-
-
-
Actions        
Stop current operation        
Put the message into the 1st spot        
Increase the priority of the message        
Decrease the priority of the message        

Implementation Plan #1 (Low hanging fruit approach)

  • Requirement #1.1 can be implemented by simply enabling AutoSyncOfflineStores option and putting the selected folder into offline mode.
  • Requirement #1.2 - skip
  • Requirement #2 - No strategy based download
  • Requirement #4 can be implemented by putting folders into offline mode, storing all operations locally, and playing them back using Playback manager (see below).
  • Requirement #5: putting folders in offline mode would store the messages locally, but the current workflow should be changed to fetch messages locally, instead of fetching them from the server every time.
  • Requirement #6 - skip
  • Requirement #7 - skip

Implementation Plan #2 (Ideal)

  • Requirement #1.1 can be implemented by simply enabling AutoSyncOfflineStores option and putting the selected folder into offline mode - adding this option into mailnews.js file.
  • Requirement #1.2 can be implemented with the help of the following new components (please note that these are conceptual rather than physical):
    • Priority Queue; Its main duty is to serialize access to imap protocol queue, and prioritize download and playback requests.
    • Message download manager; Main duties are to coordinate partial downloads, and to make strategy-based decisions to prioritize messages. In the context of this component we need to implement PARTIAL FETCH commands in imap protocol level. The existing code fetches the mime structure of the message, and looks at the types of all the parts. If a message has parts TB doesn't know how to render inline (e.g., a .zip file, or a .doc or a .pdf), it fetches the body, and the parts TB knows how to display inline separately, and doesn't fetch the parts that TB can't render inline. We can definitely leverage this feature.
    • Playback manager; To playback pending offline operations on the server and handling errors gracefully. In other word activating offline operation state machine when required. This component is partially implemented as part of Offline operation playback feature.
  • Requirement #2 same as #1.2
  • Requirement #4 can be implemented by storing all operations locally, and playing them back using Playback manager. To store the operations locally TB can be put to the following states (see Existing Behavior section):
      • Copying/Moving an IMAP message: one of the states 2, 8, 11, 12
      • Selecting an IMAP message: one of the states 2, 3, 4
      • Tagging an IMAP message: 2 or 4
      • Selecting an IMAP folder: 2 or 4 since header synchronization will be done on background - see following item
    • Common denominator of these states is 2, in other word, it is possible to cover every scenario for this feature by handling every IMAP operation as TB is in offline state.
  • Requirement #5 will be handled when #1 and #2 are implemented. Only additional requirement for this feature is to running COMPACTING operation automatically on mbox files to keep them slim.
  • Requirement #6 requires creativity to keep the local database in sync with the server. Currently TB creates fake keys for the headers generated during an offline operation, and it replaces them (the headers) during the folder update. Possible solutions are:
    1. Solving the problem at UI level as David Bienvenu once suggested "One general approach would be to have the view ignore the removal of fake keys, and the addition of new keys, and instead tell it to replace one key with an other key. But that's a pretty half-baked thought at this point. An other approach would be tell the view about a set of message-ids that this is going to happen to - then, when it gets told about a delete of message with that message-id, it would ignore it, and when it gets told about the addition of a message with that same message id, it would go find the old message with that message id, and tweak that view entry to have the new message key."
    2. Adding a new column to the local database to store the server keys when available. All operations will use local keys, server keys will be use only to sync with the server and to map header to its server counterpart.
    3. Using Message-Id header + other headers to uniquely identify messages in order to make "remove/add to the local database" decision.
    4. Implementing a middle layer to map local keys to server keys. In other word, a component to map server data model to the local one.
    • Note: Operation UNDO, and IDLE responses should be handled as well.
  • Requirement #7 can be provided by implementing new UI elements.
  • One non-functional requirement is error handling. Error mechanism should be changed. Currently all imap errors are handled in imapserver. This code should be re-factored in order to make it work with new UI elements.

Tasks for #1

  • Playback Manager - partially implemented (?, estimate: ?)
    • Activate playback state machine
    • Handle errors
    • Group playback request
  • Change existing workflow to fetch messages from the local store
  • Implement automatic compacting

Tasks for #2

  • Priority Queue implementation (?, estimate: ?)
  • Message Download Manager implementation
    • IMAP Partial Fetch command (dale, estimate: ?)
    • Local store for partially downloaded messages
    • Book keeping mechanism for message mime parts
    • Message prioritization based on attributes
    • Possibly going to touch nsImapService and nsImapProtocol components
  • Playback Manager - partially implemented (?, estimate: ?)
    • Activate playback state machine
    • Handle errors
    • Group playback request
  • Local model to server model mapping component implementation (emre, estimate: ? days)
  • Error handling mechanism implementation (?)
  • UI element implementation (?)
  • Additionally, bienvenu works on RFC 4551. He will be consulted for design, implementation and integration issues along the way.

Estimation

Feature-complete by June 22th(??). 2 weeks testing. It will be ready to ship with 3.0a2.(??).

I prefer to get Bienvenu's and Dale's opinions before coming with a realistic estimate.

Risks

[TBD]

  • Changes in event mechanism: Move, Delete, Copy event handling.
  • Unforeseen integration problems.
  • Problems in the database message key synchronization.
  • Error handling problems.
  • UI might not be ready by the deadline because of the resourcing constraints.

Decisions to make

  • Which implementation to go?
  • If we go with #2, how we keep the local data model and server data model in sync?
    • [emre] I prefer 4th approach. I can't say I totally understand the overall system design yet but it seems doable. Here is the rationale:
      • It doesn't require any change of the database schema
      • It hooks into the following points:
        1. the point where TB retrieve the headers from the server, just before writing them into the database
        2. the point where TB generates fake keys for offline operations
        3. the point where UNDOing happens
        4. just before the system exits, to write the real keys into the database
        5. it's standalone and can be tested separately
        6. (do I miss anything else?)
      • Requires no change of database implementation nor folder interaction

3) Background Message Send

[TBD]

Decisions to make

[TBD]

Retrieved from "https://wiki.mozilla.org/index.php?title=MailNews:Better_Faster_IMAP_Plan&oldid=119797"