Places:Sync
Contents
Active Work
Bugs:
- bug 374518 - Platform support
- bug 374519 - Sync client
PRD Goals
P1:
- Provide the platform-level support required for sync extensions
P2:
- Sync Client
- sync bookmarks hierarchy
- sync tag data
Proposed Firefox 3 Features
Various combinations of the following options have been proposed for Firefox 3 (roughly in order of difficulty):
- Online bookmarks backup: One-way bookmarks backup at regular intervals to a remote site of the user's choosing. This should be relatively easy to implement after we have a more modern serialization format (we'd want to move away from bookmarks.html).
- Sync service registration: Standardized UI to keep track of sync service providers and any settings required for them. i.e., account username, password, sync url, etc.
- Sync client services for extensions: Two-way services included in some fashion, to help sync extensions and promote a similar sync experience among different extensions. Probably not worth considering, except as a stepping-stone to the next feature.
- Built-in (2-way) sync: Full two-way sync with UI, likely with service registration to seamlessly support multiple services.
Requirements
Note that requirements will vary depending on the features we decide on.
- All user-defined bookmark and tag data should be serialized.
Exceptions might include very large annotations(?). Automatically generated annotations (e.g., from an extension that saves web page "screenshots" as annotations) should not be serialized for backup or sync.
- Authentication methods for backup and sync should include at least
basic and digest auth, and we should consider requiring ssl(?).
- Two-way sync should support the following scenarios:
- initial: client has data, server has none, or vice-versa
- client changes: client has changes, server has no changes
- server changes: server has changes, cient has none
- concurrent overlapping: same item changed on both client and server
- concurrent non-overlapping: different items changed on both client and server
Design
Online backup
This would consist of a service which would keep track of the locations to upload to and trigger the uploads at regular intervals. It should observe the bookmark events and supress the upload if nothing has changed since the last upload.
There are two basic variants: full and delta. In the full upload case, when the backup executes and there are changes to upload, all bookmarks are exported and uploaded. In the delta case, the client uploads a snapshot, then keeps track of the etag of the snapshot and as long as it doesn't change only uploads changes, to a separate file.
The full upload method is simpler to implement, and ensures there is always a full, ready-for-restore copy of the bookmarks on the server. However, it makes the upload operation more bandwidth-expensive.
The delta upload method is less bandwidth intensive but more complex. It would require either a "smart" server to combine the snapshot + deltas, or the client would need to be able to restore directly from the server. Manually importing snapshots + deltas is likely confusing and error-prone.
The format of the file to upload would most likely be a JSON serialization of the bookmarks data. The specific format is TBD, but would probably be exactly (?) the same format used for local backups, replacing bookmarks.html. In the delta upload case, an additional delta format would be needed.
Protocol
In order to simplify server requirements, we would limit ourselves to GET, PUT, and (optionally?) LOCK. GET and PUT are in HTTP 1.1, and LOCK is an optional WebDAV verb.
Our specific needs will depend on which flavor of online backup we wish to implement (full dump vs deltas). In the full dump case, we'd only need PUT. For deltas, we could do something like:
- LOCK the bookmarks collection
- If the bookmarks file and the deltas file haven't changed:
- GET the deltas file
- Append latest changes, PUT new deltas file
- Else:
- PUT new bookmarks file, PUT empty deltas file
- UNLOCK the bookmarks collection
Determining the latest changes requires some of the same things that sync needs. Either we must keep track of changes by logging them, or we need to keep a snapshot of the tree at the time of last sync and "diff" (e.g., see sync.js).
We could further optimize by keeping a copy of the deltas file, to avoid having to GET it before appending our changes.
UE
One possibility is to have a minimalist preference pane or dialog with something like the following:
|--------------------------------------| | [x] Enable online bookmarks backup | | | | Backup my bookmarks to this service: | | ------------- | | | Service 1 | | | | Service 2 | | | | Service 3 | | | ------------- | |--------------------------------------|
By default it would be disabled. The user would click to enable online backup and select their preferred service. This idea depends on the 'registration hooks' discussed later. An even more bare-bones approach without registration hooks could look like:
|--------------------------------------| | [x] Enable online bookmarks backup | | | | Backup my bookmarks to this service: | | -------------------------- | | | http://my.service.url/ | | | -------------------------- | |--------------------------------------|
Authentication would be prompted for when sync occurs, and the user could save their user & password in the password manager for convenience.
Additionally, we may want to add the ability to clobber the local bookmarks tree and import from the service (how?). In the delta upload case, that would be required to avoid either server smarts, or having the user download two files.
Registration hooks
This depends on what kind of information we want to save about each service. This data could be stored in the places db perhaps? At the very least it should contain a name and a url.
If we implement this, we'll also want easy ways for the user to get services added to the list. E.g., by downloading a file from the service itself which adds it to the list, much like search plugins work.
We should also consider having wizards that guide the user through setting up their backup/sync from scratch to provide a seamless experience, instead of the user having to add a service first and add folders to sync later. We'll have to square that off with some services supporting multiple shared folders.
Note that registration is desirable regardless of the level of sync support we ship with by default--one way (backup) or two way (full sync).
Two-way sync
One option is sayrer's sync.js (bug 379517). That implements a simple log-less 3-way merge. There is a sample implementation of two-way sync using sync.js in (bug 374518).
Another possibility is to log each action the user makes (by attaching observers to all the bookmarks commands). The client would keep a log of all actions and reconcile them with the server on sync. This is similar to sync.js, except that sync.js deduces the actions from the data instead of keeping track of edits.
It should also be possible to extend the "online backup" feature by optionally downloading a file with edits to the local computer after completing the upload. This would give the server the possibility to implement sync if they wish.
Open Issues
- sync all item annotations?
- security review of json on-the-wire, eval sandbox
- which external authentication mechanisms supported?
- flesh out the web service choreography
- what are the primitives? how do services specify what sync data they support?
- how do sync handlers get installed, registered, etc? new content handling framework?
local sync/access:
- ipc
- dbus
- spotlight
- google desktop
- local webserver
Related Links
- Thunderbird's roaming support is an SOC project dealing w/ vaguely similar sync scenarios.
