MailNews:Minimizing Bandwidth Usage
Make Thunderbird more friendly for unreliable and low-bandwidth conditions.
Thunderbird currently has a number of features that could help in these conditions, such as 'offline folders', 'MIME parts on-demand' and caching. But these features do not always work well in both unreliable and low-bandwidth conditions or when combined together.
This wiki pulls together inter-related issues in one place for discussion and to propose a solution.
Contents
User Experience (UX)
The desired user experience in unreliable/low-bandwidth conditions could be described as:
Only download what I ask for and only download it once.
There are a number of bugs/features at the moment that are stopping this from happening, namely bug 405437, bug 345832, bug 439731 and bug 470624 (see Related Bugs).
Desired Behaviour
1) Only download messages that I ask for
2) Once a message or part of a message (MIME part) has been downloaded it should always be stored locally:
- in offline storage if folder is __offline__ and storage rules allow (e.g. diskspace) OR
- in cache
3) A message should always be retrieved from local storage if possible.
Current Behaviour
NOTE: This refers to TB3, however currently the only significant difference is the new auto-sync feature in TB3.
1) Auto-sync (on by default) will download all messages in an __offline__ folder. We may not want this in all cases i.e. for large messages.
2) Auto-sync meets the desired behaviour in most cases (see here for more information). So lets just look at scenarios in which auto-sync is not applicable i.e. non __offline__ folders and messages that we do not want to be auto-synced.
- a) If all non-inline MIME parts < ~30KB:
- message is downloaded and stored when viewed
- b) If one or more non-inline MIME parts > ~ 30KB:
- MIME parts are downloaded on-demand and cached sometimes (there are constraints such as cache size)
3) Messages in offline store are retrieved from local storage. Cached MIME parts are retrieved from cache, however large MIME parts (i.e. attachments are rarely cached)
NOTE: Effectively this means that messages with large MIME parts (i.e. attachments) are not cached and not available offline. They have to be downloaded each time they are viewed.
Implementation
What Needs To Be Done
Although poor handling of MIME parts is a major obstacle to minimising bandwidth usage in TB it is also the most difficult to address. To get immediate results a low-hanging fruit approach is proposed, in rough order of priority:
1) Add on/off/on-demand features to auto-sync giving user full control of when sync takes place
2) User configurable auto-syncs strategies to prioritise and include/exclude messages downloaded by auto-sync
3) Better handling of MIME parts so they are only downloaded once
- a) Caching of large MIME parts e.g. attachments
- b) Support for offline storage of MIME parts
Related Bugs
Implementing these features, particularly 3a) and 3b), will have the side benefit of fixing (or partly fixing) a number of existing bugs related to messages or message parts being repeatedly downloaded:
-
bug 345832- INVALID - IMAP attachments preloading (not on demand) -
bug 439731- FIXED - Reloading of read messages when using IMAP account (enable disk cache) -
bug 405437- WFM IMAP mail are not cached for offline use if message size is bigger than mail.imap.mime_parts_on_demand_threshold - bug 470624 - described in this wiki - RFE: On-demand Auto-Sync
How To Do It
With the exception of on-demand sync 1) and 2) can be implemented using strategies. These features and more will be supported in an extension, Incommunicado.
On-demand auto-sync (bug 470624) (an oxymoron I know) however requires some development. Emre has proposed the following implementation:
- Add a new operation mode (lets say on-demand) to nsAutoSyncManager.
- Add new method to nsAutoSyncManager to explicitly tell to re-prioritize the queue.
- When running in on-demand mode, it should ignore idle events.
- When running in this mode, nsAutoSyncManager should keep downloading all pending messages until either it is all done, or the user explicitly says STOP.
Addressing 3) is non-trivial and will be tackled at a later stage, in the meantime it may be possible to avoid current problems by increasing the cache size and configuring it to save to disk rather than memory (bug 439731 #12).
On-Demand Sync
State Diagram
The finite state diagram below describes the sync states of a folder during on-demand sync. This is based on how auto-sync currently works, the notable differences are:
- The SyncCompleted and SyncFailed end states. In auto-sync a folder is simply removed from the queue either because all of its messages have been downloaded or the number of retries was exceeded.
- Substates are clearly defined. In auto-sync the substates ReadyToDiscover and ReadyToUpdate are inferred by a folder being in the Discover or Update queue respectively. In on-demand sync a folder's state will need to be explicitly defined either by a substate property or by a function that checks other attributes such as the state of internal queues.
- Folders can only be in one state at a time. This is implied by the previous point. In auto-sync a folder can be in more than one queue at a time (I think), which as an implicit state breaks the FSM model.
.---------------.
. --------------- .
|| SyncCompleted ||
' --------------- '
'---------------'
^
|
/
. discoverMessages()
| .-------.
syncCompleted() / \
. ' .
\ pendingHeaders() > 0 | | pendingMessages() > 0 downloadMessages()
------. \ .----------------. \ v .----------------. .----------------.
/ ------------- / `-> --------------- / `-> --------------- / `-> ------------------
------>|IdleCompleted| |ReadyToDiscover| |ReadyToDownload| |DownloadInProgress|
------------- <- / --------------- <- / --------------- <- / ------------------
^ \ `-----------------' `-----------------' `-----------------' ^ \
| \ pendingHeaders() = 0 pendingMessages() = 0 downloadOK() | \
/ \ / \
/ . . .
/ | | downloadFail()
/ headersOutOfDate() | |
/ | | |
. | retriesRemaing() > 0 |
| . . .
updateOK() / \ /
| | \ |
| v \ v
| ------------- ------------
| |ReadyToUpdate| |DownloadFail|
| ------------- ------------
| \ \
| \ \
| . .
. | retriesRemaing() = 0
\ updateHeaders() |
\ | .
\ . /
\ / |
\ | updateFail() retriesRemaining() = 0 v
\ v .-------------------------------------. .----------------------------------------. .------------.
---------------- / `-> ---------- / `->. ------------ .
|UpdateInProgress| |UpdateFail| || SyncFailed ||
---------------- <- / ---------- ' ------------ '
`--------------------------------------' '------------'
retriesRemaing() > 0
Milestones
Thunderbird 3.0b2:
- on-demand auto-sync (bug 470624)
- auto-sync strategy to exclude messages greater than MAX_SIZE (Incommunicado)
- on/off/on-demand auto-sync UI (Incommunicado)
Thunderbird 3.0pre1:
- save cache to disk and increase size (bug 439731 #12) (Temporary fix)
- priority auto-sync folders (Incommunicado)
- advanced on-demand auto-sync runtime dialog (Incommunicado)
- priority folders only
- select alternate strategy (e.g. sync all messages)
February/March 2009:
- support MIME parts in offline store
- always cache messages or message parts not saved in offline store
- better coordination between cache and offline store (e.g. no duplication)
Incommunicado Extension
Target Features
Target features for first release of the Incommunicado extension in rough priority order
- Simple auto-sync strategy (see nsIAutoSyncManager)
- only download messages greater than MAX_SIZE on user request
- User interface
- add column showing state of message, i.e. o - header only, x - downloaded
- icon to toggle offline/online plus support for additional states (below)
- Priority folders
- user can select which folders are highest sync priority
- Incommunicado state
- user can select 'incommunicado' in addition to online/offline
- custom auto-sync strategy
- only download messages greater than MAX_SIZE on user request
- only priority folders are downloaded
Other Concerns
Some other features/behaviour of TB that may need to be investigated to have complete control over bandwidth usage:
- Auto-update
- Junk mail
