Document Overview

Timeline:

Architecture

The Snappy Symbolication Server is a Web service for symbolicating Firefox call stacks using Breakpad symbols. It is intended to be used by the SPS profiler, the about:telemetry extension on profiling builds, and by perf-team staff for symbolicating chrome hang stacks reported to Telemetry by profiling builds.

The Symbolication Server is written in Python and runs on the breakpad-symbolapi1.dmz.phx1 Linux VM. It reads the Breakpad .sym files from a mount in the VM and caches them in memory. The .sym files contain mappings of addresses to function names for Windows and Firefox libraries. The mounts also contain Flash and other plugin symbols but these files may be removed in the near future (bug 732485). The Symbolication Server code is available on github.

The symbolication server will be publicly accessible. The symbolication requests and responses are made over HTTP in JSON format. The client HTTP requests do not carry any cookies or state. Each symbolication request consists of two fields: 1) a "stack" field with a list of stack PCs to symbolicate, and 2) a "memoryMap" field which contains information about the addresses of libraries in process memory, their PDB file name, PDB signature and PDB age. The PDB file information is used to located the corresponding .sym files on the symbol mount. The server responses contain the function names corresponding to PCs in the symbolication request.

The only data stored are the server logs. Client IP addresses, requests and responses are never logged by the server unless the server is explicitly configured for debugging.

Components

Firefox Extensions

The Firefox about:telemetry extension sends symbolication requests containing main thread call stacks and memory maps collected from transient main-thread hangs. The symbolication server will respond with symbols for each of the functions on the provided call stacks.

The SPS profiler extension collects traces from the execution of Firefox's main thread and symbolicates the traces the same way as about:telemetry.

Stored Data:

Example communication between about:telemetry extension and Symbolication Server

For a comparable memory map data example, see the memory map and stack in this bugzilla attachment.

Snappy Symbolication Server

The symbolication server parses symbolication requests and looks up function names corresponding to call stack addresses in the .SYM files on its symbol mount. The symbol files are on a read-only NFS mount to another machine on the same subnet. The symbol files contain the addresses and human-readable names for the functions in each library. The Firefox and OS symbols are stored in different directories on the mount; the server configuration file specifies where to find each.

The Snappy Symbolication Server can run as an unprivileged user.

Stored Data:

Communication with VM

Hosting VM

The VM for the server will provide a read-only NFS symbol mount to the Snappy Symbolication Server and expose a publicly accessible TCP port for client symbolication requests.

User Data Risk Minimization

In this section, the privacy champion will identify areas of user data risk and recommendations for minimizing the risk.

The risk to users' data is pretty minimal. They provide function addresses and memory maps to the server for translation into symbols that help debug crashes and similar problems. The server itself retains no metrics about the symbols translated or any individuals' patterns of stack submission.

Alignment with Privacy Operating Principles

In this section, the privacy champion will identify how the feature lines up with Mozilla's privacy operating principles.

See Also: Privacy/Roadmap_2011#Operating_Principles:

Principle: Transparency / No Surprises

This server is available to users of the SPS profiler, about:telemetry on profiling builds and by the perf-team. The users of the system need and expect to interact with Mozilla for this purpose. The server acts as a simple translating "oracle" to take care of making these symbols readable and usable for profiling.

Recommendations: none.

Principle: Real Choice

This server is available to users of the SPS profiler, about:telemetry on profiling builds and by the perf-team. Other users of Firefox don't interact with this service at all. The real choice is: users who need the system gain access to it through the profiling tools.

Recommendations: none.

Principle: Sensible Defaults

It's not contacted by default builds of Firefox. Profiling tools contact it for help with profiling.

Recommendations: none.

Principle: Limited Data

Users traffic on the system is potentially logged when the symbolication server is in debug mode. We should ensure debug mode is only used rarely and as necessary and that any server logs are collecting the minimum amount of data required and it is retained for as short a term as possible.

The Risk is that we may end up with too much logging data that never gets used for the clear value proposed to users of the system.

Recommendations: Work with infrasec and IT/Ops to minimize logging, minimize retention window, and deploy a secure logging infrastructure.

Follow-up Tasks and tracking