Identity/AttachedServices/DeploymentPlanning/FirefoxAccountsService

Overview

This is a working proposal for the backend architecture and deployment of the Firefox Accounts service.

The immediate goal of this service is to provide authentication and key/device management functionality for the New Sync project. Longer-term it may also provide authentication services for FirefoxOS, the Firefox Marketplace, and other hosted Mozilla services.

Goals and Milestones

The goal for Q32013 is to have the Firefox Accounts service Production Ready.

This does not mean having a fully-deployed production environment! With the implementation of the storage component still outstanding, there's no point in standing up an authentication service all by itself. It does mean that we need the ability to do automated deployments that pass loadtests, meet operational and security criteria, and generally inspire confidence that we could roll things out to production without major hiccups.

These are the individual milestones on our way to said goal, broken down into weekly chunks:

• Aug 09:
  • usable manual-deploy dev environment tooling
    • we will use awsbox to do simple single-box deploys
• Aug 16:
  • defined testable "success criteria" for Firefox Accounts deployment
    • Target number of concurrent users, total users.
    • Rate-per-second of various operations
      • Account creation
      • Login
      • Certificate signing
    • Target cross-region replication latency
    • Total cost of AWS resources
  • loadtesting code written and debugged.
    • this will use loads and loads.js to hook into MozSvc loadtesting cluster
    • using the client API library to script interactions with the server
    • Dependencies: stable server API and client library
• Aug 23:
  • automated single-region loadtest deployment of Firefox Accounts.
    • built using cloudformation and awsboxen
    • including a properly-configured Cassandra cluster
    • including shipping logs and performance data to a Heka/ES/Kibana box for analysis
  • loadtests run against Firefox Accounts loadtest environment.
    • Dependencies: stable and performant loads cluster infrastructure, usable log analysis tools
• Aug 30:
  • fixed any load-related issues arising from initial deployment.
• Sep 6:
  • work underway on two-region loadtest deployment
  • sanity-check whether Cassandra is working out; if not then switch back to MySQL.
• Sep 13:
  • automated two-region loadtest deployment of Firefox Accounts.
    • built using cloudformation and awsboxen
    • including a properly-configured Cassandra cluster with cross-region replication
    • including shipping logs and performance data to a master Heka/ES/Kibana box for analysis
  • loadtests run against Firefox Accounts loadtest environment.
• Sep 20:
  • fixed any load-related issues in the two-region setup.
  • tweaking, tweaking, always tweaking...
• Sep 27:
  • security review signoff.
  • svcops signoff.
• Sep 30:
  • Production Ready!

There are likely a lot of SvcOps details missing from this plan, e.g. monitoring and infrasec things. We'll do what we can to get those in place, but I'm OK with an initial loadtest environment that's missing some of that stuff.

Dev Deployment

Development deployments are done using awsbox. It is configured to stand up a single machine with all the necessary processes running. Email sending is done through Amazon SES.

Loadtest Deployment

To begin we will script this in the picl-idp repo, using awsboxen/cloudformation. That should suffice for initial QA and loadtesting purpopses. If and when we need to migrate to other SvcOps tools, the cloudformation stuff will be a good starting point.

Architecture

This will be a multi-region high-availability deployment.

We will trial Cassandra as the data-storage layer, because it promises to be easier and better than a replicated-MySQL setup. If it doesn't work out during our initial testing, we will switch to MySQL in a single-master/replicated-read-slaves architecture identical to the one used for Persona.

The Firefox Accounts service consists of four co-operating clusters of machines, based loosely on the architecture of the persona.org service:

• Webheads, which run all the read-only web views, send email, and basically do everything that's not treated specially.
• KeySigners, which accept requests from the Webheads and use master crypto keys to sign authentication tokens.
  • These are split out to separate machines since they hold sensitive private keys. Maybe we'll even HSM them or something one day.
• DBWriters, which accept proxied write requests from the Webheads and write them into the database.
  • Separating writers out into a non-web-accessible layer seems to be standard security practice for Moz properties.
• Cassandra nodes, which provides the actual data storage and replication.

Each of these clusters of machines is managed as an AutoScale group, hosted in a separate VPC subnet, and fronted by an ELB. The webheads have a public ELB, all others have a private ELB:

    client       +------------+    +-----------------+    keysign      +--------------+    +-------------------+
    requests --> | Public ELB |--->| Webhead Cluster |--- requests --->| Internal ELB |--->| KeySigner Cluster |
                 +------------+    +----+---------+--+                 +--------------+    +-------------------+
                                        |         |__
                                        | write      \__   read 
                                        | requests      \_ requests
                                        |                 \_
                                        V                   \_
                                    +------------------+      \    +--------------+    +-------------------+    cross-region replication
                                    | DBWriter Cluster |-------+-->| Internal ELB |--->| Cassandra Cluster |---------................------>
                                    +------------------+           +--------------+    +-------------------+

The use of ELBs will simplify discovery and make it easier to cope with individual instances failing. It might introduce a performance penalty though; need to test this and see!

There are no memcache nodes in this configuration. Cassandra itself has an internal in-memory cache, and they (aka The Interwebs) claim that putting memcache in front of cassandra is a waste of time. So we'll try it, profile it, and see if there is a bottleneck there that could be improved by adding another moving part.

Webhead Machines

Standard webheady stuff, running nodejs in a similar configuration to how we run Persona. Depending on costs, this will probably be a larger cluster or lighter-weight machines.

DBWriter Machines

Just like a webhead, only not directly accessible from the internet. Also running nodejs, likely in an identical configuration. Likely fewer of them, and maybe with some extra lockdown or security features.

Writes to cassandra are done at LOCAL_QUORUM consistency level, so that each region is isolated from failures in other regions. This makes us a true multi-master setup, so we need to sanity-check the data model for race conditions, lost writes, etc.

KeySigner Machines

The work done by these machines is compute-bound, so we'll probably have fewer of them running on beefier "hardware". Need to benchmark various configurations and see where the best value-for-money is.

These machines will have access to the IdP private key, which is top secret since it's the master key to the entire authentication infrastructure. So we need to consider additional security measures that could be applied. We may even want to run these on our own hardware, bridged into the VPC.

Cassandra Machines

Look up what netflix does. Do that.

rfkelly has a zany scheme for auto-bootstrapping cluster discovery, but we should use manual node allocation to start with.

These are the only truly statement machines, so they'll need to be deployed with a more puppet-like setup instead of the create-a-new-ami-for-every-change scenario used by the other machines.

Supporting Infrastructure

Each machine will run a local Heka agent to aggregate logs. They will be shipped to a stand-alone Heka router assocated with the region, which will in turn forward them to the shared ElasticSearch /Kibana thingy.

Notes from Kickoff Meeting

• Goals is to have full data replication vs. single region with separate region of "cold storage" or "hot swapability"
  • currently awsboxen deploy of full stack
  • Single master, multiple slave - as with persona
  • Cassandra can handle multi-master
  • MySQL would be single master
• Ryan is cautiously optimistic with Cassandra
  • We need a way to measure viability of both approaches
  • We need a way to measure cost of both approaches
  • OPs is backing a new approach given the PITA of maintaining multi-region MySQL cluster
• Make good use of work already completed by Gene for deploying stacks in persona Stage/Prod