Calendar:Sunbird Updates

These questions and answers are from a conversation with Paul Reed, one of the build engineers at Mozilla. The questions came from our calendar community members and I think this will help to illustrate how the update process works.

General

Automatic Update Service (AUS) is a system that updates Mozilla binaries by either supplying a complete download of a build or a partial diff from build version x to build version y. There are different channels that these updates can be dispatched to. For example, if you are on the release channel then you are dispatched only builds that have the "release" channel bit set. This means you only get the major releases. However, if you are on the "nightly" channel, then your build will update every day, once the new nightly builds are completed.

The update binaries that actually perform an update action of some sort are known as "mars" because of their file extension, ".mar". Complete updates are often referred to as "complete mars" or just "completes," and the same for "partials."

The AUS only exists for installable applications, like Sunbird, Firefox, and Thunderbird. AUS does not exist for extensions like Lightning. Lightning can be updated through the extension manager, which uses a separate update process via AMO (link). Therefore, there is no equivalent of a "nightly channel" for Lightning.

How it works

We will consider the release case here. Nightlies are done the same way, just with different business logic to worry about jumping over releases. This is necessary when someone installs build x (0.5, say) and does not catch the update to build y (0.6, say), which would be a partial. When release z (0.7, say) comes out, the 0.6 users would receive a partial, but the 0.5 users would receive a complete update. Nightly updates do this as well, but use slightly different mechanics for how they're presented to the clients.

When a release is built, the build system itself will generate the MAR files for the complete update. These are stored on FTP with the build. A complete snippet is also generated, and is uploaded to aus2-staging.m.o (TODO: get full path to new server).

A set of nasty scripts/cronjobs looks for new snippets that are arriving in aus2-staging.m.o from the tinderboxen, downloads the snippets and their related builds, diffs two complete mars and creates a partial mar, plus a partial snippet file, and pushes the partial mar to ftp.m.o and the partial snippet to aus2-staging.m.o.

When a user runs a build with a previous version, the AUS subsystem pings the AUS server in its preferences. The AUS server responds with an answer that states that either: (1) there is no update available or (2) there is an update available and it sends down the snippet so that the user can be informed about the type and nature of the update. The update is then downloaded and applied to the build.

Usually partial updates are used over complete updates in order to prevent the user from having to download a full build. If for some reason the partial update fails to apply, then a complete build is downloaded and applied in order to make certain that the user lands in a good state.

Calendar Community Questions and Answers

Here are some questions regarding AUS, Update & Build.

1. What are the differences between nightly and release update?

• None really, except for the difference in channel and business logic surrounding non-linear updates. Perhaps you can clarify this question a little?

2. What is Patcher2 needed for? Does it generate a special type of MAR file for release update?

• Patcher2 is a script that takes all the locales for a release, downloads completes for the current release and the last release, creates partials and snippets for all of these builds and puts the resulting deliverables in a state that can be uploaded back to ftp.m.o and aus2-staging.m.o. I.e. This way an Italian build won't be served an update to an English build.

3. Where are the update files for releases stored?

• Complete and partial MARs are stored on the FTP site with the build executables. Snippets are stored on the AUS staging server.

4. Do we have to change some config in Sunbird for release updates?

• You can diff the fx configs between a nightly and a release and see the difference. There are some differences in the mozconfig file that are done at build time to change items like the release channel. You can diff mozilla/tools/tinderbox-configs/firefox/ (in CVS) between the MOZILLA_1_8_BRANCH and the MOZILLA_1_8_BRANCH_release branches, the former being the nightly configs and the latter being the release configs. (TODO: Bonsai query for this diff on Sunbird)

5. Will the community AUS2 server be in place or have at least an address to configure Sunbird 0.7 in time (September/beginning of October) for the next release update (0.7 -> 0.9)?

• Yes, it should be. The community AUS2 server is a VM on on the community build VM server, and has already been created. There is a set of things that need to be done: SSL certificates, some holes punched in firewalls etc. Preed and I will take this up with IT here and work on getting these issues addressed before 0.7.

6. Public address? How do we access it? Who will maintain it? Do we have community access (-> ause?) Do we have support from #build or mozilla.org if there is an issue? For example who does security updates for the server?

• Yes, the public address will be accessible. Access to our VM will be given out to a few people in the community, including Sunbird, Seamonkey, and any others. Ause will of course be one. We need to decide who else really needs SSH access to it. We have support from the mozilla build team for build-related issues. For overall server issues, we will need to file an IT bug with the Mozilla IT team (For instance, if the entire server crashes, has an issue with a firewall etc). As far as security updates and patches to the Sunbird build VM are concerned, that is our (sunbird community) responsibility. However, it is recommended that we stay as close to the standard build reference platform so that if we need to pull the build team in to work on an issue, they are dealing with a known quantity.

7. Nightly updates (after changing the URL in Sunbird): Special configuration required on the AUS server to support nightly updates? How do we need to configure the tinderbox to upload the update information to the new server? Possibility to hook it up to <http://update-watch.localgho.st/nightly/production/latest/>?

• A plan will have to be figured out to support nightly updates moving forward. The current nightly update system may work, but there's currently disagreement in the Mozilla Build team about how to support nightly updates moving forward. Supporting/updating the current tool is an option, but a re-write may be necessary to extend the functionality to other projects.

8. Update 0.5 release to 0.7 release: We need to use the official AUS server for this. Who will take care of this? How do we create and test the update packages for all platforms x 22 locales? What tools do we need etc.? ...

• In order to upgrade users from 0.5 to 0.7 (since the AUS server changes between these two versions) we will need to have help from Mozilla for this. Here is how it will work:
  • We will post the complete/partial MARs on ftp.m.o, and then snippets on aus2.m.o to point users to 0.7. After their update, they should be pointed to the new aus2-community.m.o (name pending)
    • We will need help from preed and rhelmer for the build/server side of this, and if something goes wrong, we'll probably need to get help from sspitzer to give us a hand diagnosing issues on the client side.
  • After the upgrade to 0.7, the users will be on the new AUS server and they will be using that server from then on.
  • Ideally after this update, the users will remain on the same channel that they were on in 0.5, which for these folks will be the release channel.

9. Update 0.7 -> 0.9 release: Are the tools for package creation and testing public available so that we can do the release ourself? ...

• Partially. patcher2 is available in CVS, under mozilla/tools/patcher Preed is going to look into building a patcher configuration file that work with Sunbird. Most everything else is available or can be made available. He really needs some help to make the updates and the update process expendable to the myriad of Mozilla community projects, and if someone from the calendar community can step up to help with that work (Ause?) that would be a huge help.
• I (ctalbert) am still working on a wiki page to detail our build process so that we can easily create a build for this next release. I plan to work with Ause on this in more detail in the coming weeks. If we find that we lack access to any necessary tool or script, we'll raise that issue with the build team and get the problem addressed.