CloudServices/At a glance

From MozillaWiki
Jump to: navigation, search

TOKEN SERVER

What does it do?

This solution uses a token-based authentication system. When a user wants to connect to one of our service , they get an access token by asking a central server.

System Requirements

  • The system needs to support multiple services (not necessarily centrally)
  • The system needs to be able to assign users to different machines as a service scales out, or somehow distribute them
  • The system should consistently send a user back to the same server once they’ve been assigned
  • The system needs to give operations some level of control over how users are allocated
  • The system should provide some recourse if a particular node dies
  • The system that can handle exhaustion attacks. For example, I could set up an primary
  • The system should auto-approve any username, then loop through users until all nodes were full.
  • The system needs to support future developments like bucketed assignment
  • system that scales infinitely.

Assumptions

  • A Login Server detains the secret for all the Service Nodes for a given Service.
  • Any given webhead in a cluster can receive calls to all service nodes in the cluster.
  • The Login Server will support only BrowserID at first, but could support any authentication protocol in the future, as long as it can be done with a single call
  • All servers are time-synced
  • The expires value for a token is a fixed value per application. For example it could be 30 minutes for Sync and 2 hours for bipostal.
  • The Login Server keeps a white list of domains for BID verifications


Resources

Specs and Docs:

LOCATION

What does it do?

The Mozilla Location Service is an open service which lets devices determine their location based on network infrastructure like WiFi access points and cell towers. Project Overview

Requirements

  • Javascript
  • HTML

Get Involved

Instructions on how one can contribute to the project

MESSAGE QUEUE

What does it do?

Message Queue provides a service for applications to queue messages for its clients. Project Overview

Requirements

Notifications

The Notifications project needs Queuey for storing messages on behalf of users that want to receive them. Each user gets their own queue.

If a user needs to get notifications for general topics, the Notifications application will create a queue and clients will poll multiple queues.

The first version could be considered a Message Store rather than a queue as it supports a much richer set of query semantics and does not let public consumers remove messages. Messages can be removed via the entire queue being deleted by the App or by expiring.

Requirements:

  • Service App can create queues
  • Service App can add messages to queue
  • Messages on queues expire
  • Clients may read any queue they are aware of


Socorro

The second version allows authenticated applications to queue messages and for clients to consume them. This model allows for a worker model where jobs are added to a queue that multiple clients may be watching, and each message will be given to an individual client.

Requirements:

  • Service App can create queues, which can be partitioned
  • Service App can add messages to queue, and specify partition to retain ordering
  • Clients can ask for information about how many partitions a queue has
  • Clients may read any queue, and its partitions that they are aware of

Architecture

Queue Web Application 

   queuey (Python web application providing RESTful API)

Queue Storage Backend 

   Cassandra

Coordination Service 

   (Only used when consuming from the queue)

   Apache Zookeeper

Queue Consumption 

   * Consumers coordinate with coordination service
   * Consumers split queues/partitions amongst themselves
   * Consumers record in coordination service the farthest they've processed
     in a queue/partition
   * Consumers rebalance queue/partition allocation when consumers are added or
     removed using coordination service
   * Consumption and rebalancing is done entirely client-side

Queuey is composed of two core parts:

  • A web application (handles the RESTful API)
  • Storage back-end (used by the web application to store queues/messages)

The storage back-end is pluggable, and the current default storage back-end is Cassandra.

Unlike other MQ products, Queuey does not hand out messages amongst all clients reading from the same queue. Every client is responsible for tracking how far it has read into the queue, if only a single client should see a message, then the queue should be partitioned, and clients should decide who reads which partition of the queue.

Different performance and message guarantee characteristics can be configured by changing the deployment strategy and Cassandra replication and read / write options. Therefore multiple Queuey deployments will be necessary, and Apps should use the deployment with the operational characteristics desired.

Since messages are stored and read by timestamp from Cassandra, and Cassandra only has eventual consistency, there is an enforced delay in how soon a message is available for reading to ensure a consistent picture of the queue. This helps ensure that written messages will show up when reading the queue to avoid 'losing' messages by reading past where they appeared in the queue.

For more on what kind of probabilities are involved in various replication factors and differing read/write CL's, see:

In the event that the clients are deleting messages in their queue as they've been read, the delay is unimportant. Enforcing a proper delay is only required when clients read but never delete messages (and thus track how far into the queue they've read based on time-stamp).

When using queue's that are to be consumed, they must be declared up-front as a partitioned queue. The amount of partitions should also be specified, and new messages will be randomly partitioned. If messages should be processed in order, they can be inserted into a single partition to enforce ordering. All messages that are randomly partitioned should be considered loosely ordered.

Queue Workers

A worker library called Qdo handles coordinating and processing messages off queues in a job processing setup. The Qdo library utilizes Apache Zookeeper for worker and queue coordination.

Workers coordinate to divide up the queue's and partitions in each queue so that no queue/partition has multiple readers. This avoids the need for read locking a queue, and how far into each host+queue+partition is stored in Zookeeper.

This model is based exactly on how Apache Kafka workers divide up queues to work on.

API

API can be found on the queuey API docs page.

Engineers

  • Ben Bangert
  • Hanno Schlichting

NODE ASSIGNMENT

What does it do?

The Node Assignment product provides a central server to allocate users of Mozilla Services products to a node associated with that product. It is a standalone service (though of minimal use outside of the company or with no other products to support)

Project Overview

APIs

This service incorporates two APIs:

Internal

An internal API for node management - addition of nodes, adjustment of weighting factors, downing and backing off nodes

There is no authentication attached to the internal API. It is assumed that it will only be accessible locally, and with any protections provided by the server.

All API responses return json.

GET

/{product} - returns a list of all clusters in the product
/{product}/{cluster} - returns a hash keyed by nodes in the cluster. Values are hashes of the data for each node.
/{product}/{cluster}/{node} - Returns the hash of the data for a single node.

PUT

/{product}/{cluster}/{node}/{key}
/{product}/{cluster}/{key}
/{product}/{key}

Sets the {key} field to the value specified in the PUT body. The scope of this change depends on whether {cluster} (and {node}) are specified in the URL.

Valid keys are: weight, current_in_period, down, backoff. Attempts to set any other keys will get a 400 error. Those should be set using a script or directly into the DB.

A successful PUT will receive a 0 as the response

External

The external API has one function call, and is password-protected by central auth:

/<version>/<username>/<product>

(it's assumed that the DNS entry point will be solely for this service. If this doesn't appear to be correct, we'll need a /prefix at the beginning.)

This API call will return one of the following:

503: internal error
401: <username> fails to auth
404: <product> does not exist (or URL doesn't exist entirely. Clients should generally interpret this to mean use the same server to which this query was issued.
200 ('null'): no node for the product is available for assignment
200 (other text): the name of the node, including protocol, that has been assigned to the user

Design

The user's nodes will be stored in the authentication LDAP, in an array under the primaryNode attribute. The storage format will be node<internal information. Internal information consists of arbitrary data, as defined by the product.

In a situation where the user already has an assignment corresponding to the product, authenticating the user will be sufficient, as the data from that will be sufficient to fulfill the response.

If there is no record for the product in the primaryNode field, the application will request an assignment from the assignment table. This is a mysql table that will evaluate the available nodes for a product, and produce the one with the best current availability, returning the data to be written into LDAP.

DB Fields

Product - Name of the associated service
Cluster - Cluster for the service. This will generally correspond to the colo, but we may end up with multiple clusters in the same colo
Node - The specific node in the cluster that a user is assigned to
LDAP - The value to be written into the LDAP if this node is selected
Available - A count of remaining assignments available in this period. This value is not defined by nodeassigment and is simply set by the service at whatever period is desired.
Current Load - Although it is not a direct reflection of users, it will be incremented by 1 for each user added. Services should set their weights according to how much impact they want each additional user to have to the algorithm
Capacity - The theoretical maximum weight to be associated with the node
Down - a binary value indicating whether a node is down. The node service makes this available via api, but it is expected that the service itself will be regularly checking the value. Users will not be assigned to a node that has been marked as down.
Backoff - a value in seconds that should be added to responses from services that support backoff. As with the down flag, services are responsble for accessing the data here and pushing it to their machines.

Assignment Weighting

When a request comes in for a new node, the api will choose the available server with the lowest Weight/Capacity number, then increase the weight by 1


What This Isn't

Config Management: We should have central config management for the various node configurations for the various services. The first attempt at a node-assignment service included that, but this version does not. There are several reasons for that:

  • It should be handled by operations, who may not want to do it as a service, and almost certainly don't want it to depend on another service. If a service is needed to make this work, it can be written separately
  • Not including this functionality here means we don't need to expose our config database to an external service, even if it has layers of indirection
  • The different services are likely to have a wide variety of config requirements, which would complicate an otherwise straightforward DB.
Retrieved from "https://wiki.mozilla.org/index.php?title=CloudServices/At_a_glance&oldid=1109499"