Penelope Developer Page

Building Penelope

Start to become familiar with the process by building Thunderbird. Go fetch and build Thunderbird. Start learning how it works.

Linux

These were the steps followed to build Thunderbird 3.0 and Penelope from comm-central on Linux. These instructions assume an up-to-date Linux machine. The machine I used is running Gentoo Linux 2006.1. Note that using the latest versions of OSes can be problematic due to dev libraries that are not compatible with older runtime versions. For example, Ubuntu 9.10 includes version 2.18 of libgtk which is incompatible with older runtime versions, and so it is recommended to use Ubuntu 8.04 LTS for a build environment.

More Thunderbird Linux build details can be found at developer.mozilla.org

• Be sure that your umask is set to 022. If it's not then lots of things will break. You need to set it to 022 before you grab the source.

• Create a new directory, call it src

• hg clone http://hg.mozilla.org/comm-central/ src

• cd src

• python client.py checkout

• cd mozilla/extensions

• mkdir penelope

• hg clone http://hg.mozilla.org/penelope/ penelope

• Create ~/.mozconfig with:

   mk_add_options MOZ_MAKE_FLAGS=-j4
   mk_add_options MOZ_MAKE_FLAGS=-w
   mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../bin/debug
   
   ac_add_options --enable-application=mail
   
   ac_add_options --enable-default-toolkit=gtk2
   ac_add_options --enable-xft
   
   ac_add_options --disable-optimize
   ac_add_options --enable-debug
   ac_add_options --disable-static --enable-shared
   ac_add_options --disable-installer
   ac_add_options --disable-tests
   
   ac_add_options --enable-extensions=default,penelope,inspector

Note that it is this last line that includes the building of the Penelope extension in the main build. You don't technically need the "inspector" (DOM Inspector) extension, but it's nice to have if you need to debug DOM issues.

• make -f client.mk build

• Go get breakfast/lunch/dinner because it takes a while to do a full compile.

OS X

The steps to build Penelope from comm-central on OS X are almost identical to the instructions for Linux. There are may caveats depending on whether you want a universal binary, which version of OS X you have, which version you want to target, etc. More Thunderbird OS X build details can be found at developer.mozilla.org.

The only deviation from the Linux directions is the installation of LibIDL and GLib, and contents of the ~/.mozconfig file.

• Install LibIDL (via orbit) and GLib using fink:

   $ sudo apt-get update
   $ sudo apt-get install orbit orbit-dev glib

• Create ~/.mozconfig with:

   mk_add_options MOZ_MAKE_FLAGS=-j4
   mk_add_options MOZ_MAKE_FLAGS=-w
   mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../bin/debug
   
   ac_add_options --enable-application=mail
   
   ac_add_options --with-macos-sdk=/Developer/SDKs/MacOSX10.4u.sdk
   ac_add_options ppc --enable-prebinding
   
   ac_add_options --disable-optimize
   ac_add_options --enable-debug
   ac_add_options --disable-static --enable-shared
   ac_add_options --disable-installer
   ac_add_options --disable-tests
   
   ac_add_options --enable-extensions=default,penelope,inspector

Making universal binary builds on OS X

Universal binaries are executables for Mac OSX that contain both PowerPC and Intel processor binaries. That way you can distribute just one application and installer that will work on both types of processors.

Be sure that you can successfully build for just one processor using the steps above. That way you won't be confused if a problem arises and not know whether the issue is with building in general, or with universal binaries in particular.

The Mozilla Developer center has some documentation that you will need to follow. In particular there are a couple of things worth pointing out as they can cause problems if not set up correctly:

• Make sure you have at least version 2.4.1 of Xcode due to a bug in Xcode 2.4. You have to be a Apple Developer Connection member, but that's free and only requires a valid email address.
• Install the 10.4 SDK from the Xcode disk image. If you've already installed Xcode, then you can easily add the 10.4 SDK by running the CrossDevelopment.mpkg package from the Packages folder of the Xcode disk image. The 10.4 SDK doesn't get installed by default, so you'll have to do a "Customize" install and select the 10.4 SDK.
• You must use the MOZ_OBJDIR option in your .mozconfig file in order to place the generated files outside of your mozilla source tree. This is because both PPC and Intel binaries need to be created and must be kept separate. The build process will create two directories (ppc and i386) in your MOZ_OBJDIR directory, and the universal build will be placed in ppc/dist/universal with a symbolic link at i386/dist/universal.
• Last you need to add these lines to your .mozconfig:

   if test -e "$topsrcdir/mozilla/build/macosx/universal/mozconfig"; then
   . $topsrcdir/mozilla/build/macosx/universal/mozconfig
   else
   . $topsrcdir/build/macosx/universal/mozconfig
   fi

Windows

See the steps above under Linux for how to get the source code via comm-central.

Penelope is back to using the trunk version of the Mozilla platform, which uses the MozillaBuild environment. Make sure you go through in detail the setup documentation. It's pretty simple to set up with MozillaBuild now. You just download the package and the installer gives you all the right versions of the tools you need to build. You have to use their MINGW32 shell (which is basically bash) in order for it to compile correctly. It's easiest just to use their batch files for starting up the shell and getting the environment set up right. I've only tried it with Visual Studio 2005 (the official version of VS for the trunk), which has a corresponding batch file of start-msvc8.bat.

Here's the options for your ~/.mozconfig file in order to make a debug build:

   mk_add_options MOZ_MAKE_FLAGS=-j4
   mk_add_options MOZ_MAKE_FLAGS=-w
   mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../bin/debug
   
   ac_add_options --enable-application=mail
   
   ac_add_options --disable-optimize 
   ac_add_options --enable-debug
   ac_add_options --disable-static --enable-shared
   ac_add_options --disable-installer
   ac_add_options --disable-tests
   
   ac_add_options --enable-extensions=default,penelope,inspector

Finally, go to the root of the source tree and do a:

• make -f client.mk build

For some reason the executable that winds up being built can't be launched from a MINGW32 shell. If you try to launch it it fails to initialize some components and gives up. You have to launch it from a different shell to get it to start up properly.

Windows Vista

Windows Vista provides some roadblocks in compiling Penelope. The first is the new security model. By default, administrator accounts under Vista run applications at a reduced privilege level called Standard User. Microsoft found that the most common legitimate reason to need administrator rights was when installing an application, and in order for all the existing installation programs to work under Vista they had to come up with a heuristic workaround. What Vista does is automatically attempt to elevate the privilege of any application that has the words "install" or "setup" in the name of the executable filename. I say "attempt" because there appear to be some situations where the privilege elevation fails, and one of those happens to be when called inside of scripts. I think it has to do with the way the process is created (privilege elevation happens when you call ShellExecute(), but not when CreateProcess() is used), but I'm not sure of that entirely because I don't know how MINGW32 shells start up new processes.

And wouldn't you know it, one of the utilities used to compile Penelope falls in to this heuristic: nsinstall.exe. All that command-line tool really does is copy some files, which doesn't require Administrator privileges as long as you have have write access to the destination directory, but Vista's automatic privilege elevation thinks it might due to the name. The privilege elevation fails and so the build fails. Happy, happy, joy, joy.

There's two current workarounds. One is that the Command Prompt window that you use to compile Penelope can be run as Administrator. You can create a shortcut setting that up (its in the Properties, Compatibility tab), or you can right-click on a link to a Command Prompt (e.g. one that shows up on the Start menu) and select "Run as Administrator". This is an unsatisfying solution because it raises the privilege level unnecessarily, but it does work. It does happen to solve other permission-related issues with files in general, so it is an easy way to fix the problem. It's also no different than the security model of Windows before Vista (well, at least the way that most people set it up, which is to run as Administrator all of the time).

The other way is to tell Vista that the app doesn't really need Administrator privileges. You can do this via a manifest file, which is an XML file which has properties about the application. It has the same name as the executable (including the ".exe"), but with an extra extension of ".manifest" to it. The manifest file contents should look like this:

   <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
   <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
     <assemblyIdentity version="1.0.0.0"
        processorArchitecture="X86"
        name="nsinstall"
        type="win32"/>
   
   
     <description>Description of your application</description>
     <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
       <security>
         <requestedPrivileges>
           <requestedExecutionLevel
             level="asInvoker"
             uiAccess="false"/>
           </requestedPrivileges>
          </security>
     </trustInfo>
   </assembly>

The pertinent part that makes it work is the level="asInvoker" attribute, which tells Vista that it doesn't need any extra privileges to run. There's one snag to this, which is that if you've already run nsinstall.exe without the .manifest file, then Vista has already put the perceived needed privilege elevation in the fusion cache. In order to clear the cache you need to reboot Vista.

There is a third way to workaround the problem, but I don't like the long-term implications of it. You could rename the nsinstall.exe utility to something that doesn't have "install" or "setup" in the name and then change the build scripts to that effect. However, that means keeping a branch of the build scripts forever and that would probably cause more headaches down the road.

Reducing Build Time

The Thunderbird folks advise building in a subdirectory or subdirectories when making a change to the code to reduce the amount of time needed to build. By a narrowing down the build to at least the component you need to rebuild you can cut the compile time dramatically. You can cut down the build time even further by specifically providing the change logic yourself and rebuilding only those portions of the code that you know you changed.

If you use a separate object directory (as the default config suggests), then you'll need to look for the individual makefiles in the object directory hierarchy (for example in my setup I found them under "thunderbird-static" since that's my object directory name). Otherwise the makefiles will be alongside the source.
Scott MacGregor notes:

I don't use the objdir technique, but some developers really like it.
Most of those folks like to use it because they are building multiple
products like firefox and thunderbird and they only want to have one
source tree directory. They can just point their thunderbird build at
one objdir directory and the fx build at another, while working out of
just one instance of the source tree. But I mostly just build thunderbird,
so moving the objdir out of my source tree doesn't help me much.

Specific Example

A lot of the core functionality is provided in mail.dll, which is built from the mailnews directory.

For example, say I wanted to make a change to a portion of the IMAP code (e.g. I want to modify "nsIMAPNamespace.cpp").

Slow Method

The really slow method to recompile would be to invoke:

make -f client.mk build

in the root level of the mozilla source code. On my laptop, even with no changes to the source code, this takes over 12 minutes to complete.

Faster Method

A faster way to recompile would be to invoke "make" inside the mailnews directory (no arguments are needed for "make" since the mailnews directory contains "Makefile"). Without an object directory specified that would be mozilla/mailnews; with an object directory specified that would be something like mozilla/objectdir/mailnews.
On my laptop, this took less than 2 minutes to recompile.

Fastest Method

The fastest way to recompile would be to first invoke "make" inside the mailnews/imap directory.
Then invoke "make" inside the mailnews/build directory (which just re-links mail.dll).
On my laptop, this took less than 10 seconds to recompile (ignoring time for me to type, change directories, invoke the 2nd command etc.)

Troubleshooting

Here are some problems that have cropped up during development:

• The prefs.js file in the profile directory may have stale settings in it which override the behavior you are trying to test. You must remove them manually.
  • On Linux the profile directory is in ~/.thunderbird/
  • On Mac OSX the profile directory is in ~/Library/Thunderbird/Profiles/
  • On windows the profile directory is C:\Documents and Settings\<Windows login/user name>\Application Data\Thunderbird\Profiles\
• The extensions subdirectory may contain the penelope extension if it was installed for Thunderbird. This will conflict with an install of Eudora because the penelope extension resides in the Eudora install directory. Both extensions will be applied at the same time which will cause problems. If you install Eudora, you must remove any previous install from the profile directory
• Beware cross-volume symlinks (see Discussion page)
• Very bad things happen when you have an unresolved entity (missing a DTD file or entry in a DTD file). This can manifest as the whole application is a window with zero size (which is very hard to find!).
• Multiple key mappings occur when they are scoped in different keysets. When redefining a key mapping, make sure it is done in the correct keyset ID.
• Some key mappings canot be set via the prefs file. They seem to be dynamically updated via a xul overlay at run time. The only way I have found to redefine these keys is to create an overlay that remaps the keys.
• If the resulting build in dist/bin will not run try adding execute permission to dist/bin/run-mozilla.sh

• To re-create the Makefiles you have to remove the generated configure files then re-run make:

   rm config.log config.cache config.status mozilla-config.h
   make -f client.mk build

Debugging the Build Process

Trying to figure out what went wrong with a build can be tricky. Here's a tip for helping troubleshoot builds.

The -w flag will output a diagnostic line every time you enter and leave a directory. So it gives you a context of where the following commands are coming from. It also gives a nesting level. This flag is on by default with the Cygwin version of make on Windows, but needs to be added for Mac and Linux. To add it, either add this line to your .mozconfig:

   mk_add_options MOZ_MAKE_FLAGS=-w

or set the MAKEFLAGS environment variable to be -w.

Building Installers

Once you're done building the application, you also have the option of building the installer. Instructions are here, but they basically just say to cd in to MOZ_OBJDIR/mail/installer, and do a make from there (or make installer for Windows). Also for Windows, the instructions note there are some extra tools needed to build the self-extracting installer. For Mac universal binary builds, do the make from MOZ_OBJDIR/ppc/mail/installer. After building it, the Windows installer can be found in MOZ_OBJDIR/dist/install/sea/. The Mac disk image will be in MOZ_OBJDIR/ppc/dist/. The Linux tarball will be in MOZ_OBJDIR/dist/.

Windows Issues

Since the Windows versions are built with the Visual C++ 8 compiler, the C runtime DLLs need to be present in order for the application to run. So the installer needs to be told to include those DLLs, which is talked about in Bug 324690. Basically you just need add the environment variable WIN32_REDIST_DIR, and set it to the directory in MSVC that holds the Win32 VC redistributables, which if you installed VC in the default location would be C:\Program Files\Microsoft Visual Studio 8\VC\redist\x86\Microsoft.VC80.CRT. It's interesting to note that the installer will place those DLLs in the Eudora install directory, rather than in Windows system directory. It's easier to do it that way, and has the advantage that those DLLs won't interfere with any other apps. The disadvantages are that it takes up more disk space and that the DLL code space isn't shared between multiple running apps.

Also, in order to correctly label the build for distribution there are a couple of environment variables that need to be added:

export MOZILLA_OFFICIAL=1
export BUILD_OFFICIAL=1

More info can be found here. There's some extra stuff on that page that doesn't really need to be done, like adding the --without-system-* options isn't necessary since we already specify the --enable-static option.

Customizing Mac Disk Images

The Mac installer is a disk image. When you open it, it mounts a volume that holds the application. It first shows a license agreement (EULA) and requires the user to agree to the license before proceeding. It then shows a folder containing the application, and the user needs to drag the application to the Applications folder in order to install it. The appearance of that folder is controlled by the .DS_Store file in the folder, just like all folders in the Finder. .DS_Store holds the window position and size, the size of icons, the arrangement of the files in the folder, the background image, and other properties. To modify these properties you just have to modify the .DS_Store file.

There are some web sites out there that mention you can create a dummy disk image, muck with the folder window in it, and the save the resulting .DS_Store file to reuse in another disk image. Turns out that for compatibility on other machines this doesn't always work. Evidently the Finder can create .DS_Store files that will not work when moved to a different machine.

What the Mozilla platform uses is a .DS_Store file that was edited by hand and known to be compatible. This file is named dsstore and is kept along with the branding. The build process for the disk image copies $(DIST)/branding/dsstore in to the disk image as .DS_Store.

Mark Mentovai is the developer who came up with the Mozilla disk image process, and he reverse-engineered the .DS_Store file format to a fairly complete degree. His notes for that can be found here. It's very helpful for figuring out where to make manual changes in the dsstore file.

Note that if you want to change the background image all you have to do is check in a new background.png file, and no changes to the dsstore file will be necessary. The most likely changes someone will need to make to the dsstore file in the future will be to window size and/or location.

Development Help

Web Sites

There are a number of web sites out there for learning about the Mozilla platform and technologies. Here are some that may be helpful.

The Mozilla Developer Center contains lots of articles and references on all things Mozilla. In particular the section on XULis useful for extension developers.

XUL Planet has full references for XUL, DOM, and XPCOM, as well as tutorials and guides.

Bonsai is a web query mechanism in to the Mozilla CVS source code depot. Here's a query that will show you all Penelope changes.

The Mozilla Cross Reference (or mxr) provides both an indexed search engine and a source browser to the Mozilla code base. There's also an hg interface in to both comm-central and Penelope which you can use to browse through source files.

Tools

Here's some downloadable tools you can install to aide in the development process.

The Venkman JavaScript debugger is essential as a debugging tool to step through JavaScript code and look at variables while running. For more info see the Venkman page on MDC.

DOM Inspector allows you to browse and edit the document object model tree of UI elements. You can download it here, but it only works for 2.0 versions of Thunderbird, not trunk versions. If you are building Thunderbird yourself you can build a trunk-compatible version of DOM Inspector by including inspector in the --enable-extensions option (look here for exact syntax).

Chrome List lets you browse the chrome in an app.

The Extension Developer's Extension holds a number of handy developer tools.

Deconstructing Mozilla

Adding a New Menu Command
Address Book
Filters
IMAP
Importers
Preferences
Thread Handling