Background Updates/Implementation Strategies

Implementation Strategies

Side-by-side approach

This is the approach where Firefox would download an update in the background, apply and install it alongside with the existing installation, and would then wait for the next startup. At startup, the firefox.exe stub would look for the existing installation directories, and would pick the newest one, and start up either by loading xul.dll and the rest of the libraries from that directory or by running a firefox.exe from that directory. After startup, when the next update arrives, the older version (which is not in use any more) will first be removed, then the new update will be applied and it will get picked the next time that Firefox starts up.

LawrenceMandel The way I read this there will be two copies of Firefox on the system during an update: the currently running Firefox and a completely new, updated Firefox. Is this correct? Will this require twice the disk space? How much disk space are we talking about? Is this a concern?

(ehsan) Yes, we're talking about having two copies of Firefox on a system. This would require twice the amount of disk space (although we could resort to hardlinking on platforms which support it to reduce this). It would be an issue for mobile, except that mobile is completely out of scope here. I'm not sure if this is a concern with Desktop Firefox.

LawrenceMandel Yes. Mobile is out of scope as updates are handled via a different process. (App store.) I hope that disk space (~75Mb on Mac) isn't an issue but I don't know. Could Firefox ever get into a state (say if a user has several updates to perform and doesn't reboot) where several updates have been installed and are hanging around on a system so that Firefox is eating up 300, 400, 500+ Mb of disk space at a time?

(ehsan) Our system requirements page says that users need a minimum of 200MB of disk space, so I think we're safe there. The way that side-by-side approach is designed, you'll only ever have two directories: currentVersion and nextVersion. If a new version becomes available when there is already a nextVersion directory, we will just remove the old nextVersion directory and write a new one.

Replacement approach

This is the approach where Firefox would download an update in the background, apply and install it in a new directory, and then do one of the two things depending on the OS:

If the OS supports removing files which are currently in use (*nix), it would just remove the existing installation and move the new directory in place of the old one.
If the OS doesn't support removing files which are currently in use (Windows), the old directory should be marked to be removed during the next reboot and Firefox should check at start up to see whether the previous directory is in use, and if not it should move the new directory in place of the old directory and then restart.

LawrenceMandel Do you mean reboot the OS or reboot Firefox? (I hope the latter.)

(ehsan) Reboot the OS. The user doesn't actually need to restart their system, the files will just stick around until that they do. FWIW, this is how all applications remove files which are currently in use on Windows.

LawrenceMandel Perhaps a naive question but...in the case where Firefox is updated and restarted, the newly updated Firefox will be running. Will the old one still have files that the system considers "in use" that prevents them from being removed until a system reboot?

(ehsan) It might (in case the old version was not killed correctly). It might also be the case that another application is keeping the files open for some reason. But when I talked about removing the files on next reboot, we need to do that since we don't have any way of knowing if we can run any code after Firefox shuts down (i.e. we don't know if the user will run Firefox again).

LawrenceMandel Eclipse's bundle mechanism supports the ability to register and unregister services with the software system. In this way an update can be applied to a running system without restarting the system. Even if the core executable required a different mechanism this may provide benefit as Firefox would not need to be restarted to update. Is there any way that we can update components of the Firefox system without restarting the system?

(ehsan) Not easily. We'd basically need to redesign everything from the ground up with that in mind. I think that's outside of the scope of this project (and to be honest I'm not sure if it's feasible).

LawrenceMandel That was my expectation but thought I would ask anyway.

Challenges

The main challenge in this work is that on Windows, files which are currently in use cannot be replaced. So we cannot really update Firefox as it's running. The side-by-side approach avoids this problem by guaranteeing that files which are in use are never attempted to be overwritten (see below for the only exception to this claim). The replacement approach doesn't avoid this problem completely, so instead it relies on working around the problem during startup. This may or may not have negative impact on startup performance.

Stub executable

The side-by-side approach requires a stub executable in order to work correctly. We effectively have this executable, which is the firefox.exe program itself. There is very little code in this file compared to the rest of Firefox living in shared libraries, but we need to have a solution for the case where we need to update this executable (because in the side-by-side approach, the stub does not usually change.)

One approach in updating the stub executable is for it to check inside the library directory that it decides to load libraries from and look for a firefox.exe executable. If one is found there, it indicates that the stub should be updated, so then we can use the current trick of updating update.exe (writing a copy of the stub to another directory, launching that one for the sole purpose of replacing the stub, and then restarting using the new stub). This additional operation only needs to be performed when the stub is actually changed during the update. The updater can check and make sure that the stub is changed when it's applying the update in the background, and remove it (or don't write it to disk in the first place) if it has not changed.

Details on the side-by-side approach

Versioning

We need a way to name the two versions living next to each other. My proposal is to use the following format

<hg repository index>.<hg full revision ID>

("hg repository index" is the number of the changesets living in the hg database, which is seen throughout the hg UI).

The rationale for this is that we need a way to determine which one of two versions is newer, and coming up with a method for that is really tricky in a non-linear multi-repository development model, such as what we have now. Here are a number of use cases which is satisfied by this scheme:

The comparison is simple (just a numerical comparison) and doesn't rely on filesystem timestamps.
The comparison does not rely on Firefox version numbers, which makes it work for branches which update with the same version number (such as Nightly)
We can use the hg full revision ID in the updater later to refuse to install two versions with the same repository index and the same hg full revision ID. If the repository index are the same and the revision IDs are not, then we can artificially increment the repository index of the update to be installed by 1 so that it gets picked up later in the startup process.
The comparison works with our current multi-repository release model. Under all conceivable circumstances, we can assume that if F returns the number of revisions in an hg repository, then:

F(mozilla-release) < F(mozilla-aurora) < F(mozilla-beta) < F(mozilla-central)

LawrenceMandel Is there a build id/timestamp? Is that a suitable identifier to use? Would this method break if FF moved off of hg?

(ehsan) There is a build ID, but using that would fail for updating across branches. I don't really want to use timestamps since they are prone to all sorts of race conditions, and also to the correctness of the clock on our builders/users' machines. I don't know of any other suitable identifiers. Moving off from hg would break this unless we come up with a way to get a repository index from the repository. I would imagine that any RCS that we would use would have to have a similar concept. Since we only use the revision ID for uniqueness, it's treated as an opaque value so switching away from hg won't affect it.

Downsides of this approach:

Updating between development branches might not work correctly. For example, if we ever decide to move a bunch of users from the nightly update channel to the devtools update channel, the update process might fail to work because the devtools branch might have fewer chanegsets in the Mercurial repository.

LawrenceMandel Perhaps we need to have a product specific preference included in the version. For example, ffn.<hg repository index>.<hg full revision ID>, ffa.<hg repository index>.<hg full revision ID>, ffb.<hg repository index>.<hg full revision ID>, ff.<hg repository index>.<hg full revision ID>. In this way you can have different streams installed concurrently. Alternatively, if we already install different streams into different directories that approach can be expanded to support dev tools or other similar 'like' builds of FF.

(ehsan) Right. We should see if this is a use case that we want to support first though.

Migration

We need a migration plan so that we can get users from the current installation model to a side-by-side installation model. For example, on Windows, here's what we currently have in an installation:

C:\Program Files\Mozilla Firefox\
                                 firefox.exe
                                 xul.dll
                                 ...
                                 components\
                                            ...

This is what the target will look like:

C:\Program Files\Mozilla Firefox\
                                 firefox.exe
                                 77562.45ac18984273\
                                                    xul.dll
                                                    ...
                                                    components\
                                                    ...

A similar directory structure is proposed for Linux and Mac as well.

LawrenceMandel The example above adds 19 characters to the path. Is there any risk of hitting the Windows path limit of 255 chars due to this addition?

(ehsan) Interesting point. There's very little risk in a standard installation. But it is possible to exceed that limit in the pathological cases. We might need to protect against that in the updater somehow.

Here's the plan of attack for this problem:

Make the stub executable aware of this directory layout.
- The installer will look for directories named \d+\.[a-fA-F\d]{40}.
- If more than such a directory is found, the stub executable will do a version comparison and pick the newer one.
- If such a directory is found, the stub executable will load DLLs from that directory instead of its own. Platform specific details TBD.
- If such a directory is not found, everything will proceed as it does today.
- If a version specific directory is found, that will serve as the process directory for all intents and purposes from that point on.
Change the packaging step to actually package stuff under that structure.

Mossop The 255 limit only applies to certain APIs and it is possible to support much longer path lengths. I would love it if we could make that work throughout XPCOM since we hit it occasionally when users have long usernames. Probably out of scope for this though.

Alternative approach

I'm not yet quite confident that it's OK to have a current directory without firefox.exe inside it. An alternative approach is to write a new stub executable from scratch which does as little as possible, picks the latest version and forks the firefox.exe process inside that directory. That way, almost nothing will change compared to the way existing installations work, except for one component being added to the path.

Mossop Lots of stuff uses the current process directory as a base to search and find files in the app directory. Keeping that the same would vastly reduce the amount of changes necessary and make this safer since we probably don't have great test coverage for some of it.

Downgrade path

We need to have a clear solution for users which install an older Firefox version without the side-by-side installation feature. I believe that the installer should just look at the log of the previous installation and just remove it before doing a new installation. The only tricky question is the name of the version directory. I think this may be a problem in the following scenario.

User installs Firefox 10.
User upgrades to Firefox 11. Now, the path names inside uninstall.log point to the side-by-side installation for 10.
User reinstalls Firefox 10 over the existing version. The 10 installation directory is removed by the uninstall process, and Firefox 10 is reinstalled in the same side-by-side directory.
User starts up Firefox. The stub executable will see side-by-side directories for 10 and 11, and will pick 11 and starts that up.

This will effectively break installing older versions over newer versions.

LawrenceMandel How do we handle downgrading today? Can a user install an older version on top of a newer version? Do they have to uninstall before reinstalling the older version?

(ehsan) The installer uninstalls the previous version first, so yes, this will work today.

Risks

What happens to the applications which install stuff to the firefox installation directory? (Does that even work these days?)

Mossop Extensions, plugins and I think searchplugins might be affected here. The extensions case I don't care about, third-parties shouldn't be sticking their stuff into the app directory. The plugins case I think there are preferred alternatives as well but should check with Josh maybe. Search plugins I am unsure about, Gavin might know more.