CloudServices/Sagrada/Roadmaps/2011Q3

Introduction

Services has passed several milestones since its separation as a team from Labs. It has launched multiple products, successfully transitioned the flagship product (Sync) to a new codebase, worked with ops to build a robust operational infrastructure, and supported the rapid user growth that comes from being integrated into Firefox.

Up to this point (mid-2011), the focus has been on launching these products. It's been done well; things are generally modular and focused on code reuse, but the present allows us an opportunity to step back and focus on building a platform for further applications. By analyzing these as generic components, rather than code pieces working towards a goal, we can make them robust enough to support anticipated future work and flexible enough to accommodate reasonable unplanned feature needs. These components should be available to all of Mozilla (as well as the open-source community) to enable rapid development of services. The fact that this is a framework focused on REST-like *services* is what sets us apart from the other frameworks out there.

Q3 will be focused on cleanup and getting these tools into shape, as detailed below.

bug 676423 - Master bug to track progress of the other bugs.

Core Components

Baseapp [GREEN]

It should be a goal of all services projects (with the possible, though unlikely, exception of AP) to make use of the baseapp with as little overriding as possible. This means that we need to have the ability to load arbitrary classes defined in the config file into the app namespace. How can we make building webservices easier?

Bugs:

• bug 672664 - Allow configuration to load modules (including auth) into env.modules dictionary - DONE
• bug 674768 - Turn authentication off by default - DONE
• bug 625183 - Configuration to bail on bad XFF header
• bug 670630 - Support for HEAD __heartbeat__ - DONE
• bug 668664 - Make sure pending queries are killed on app restart - WAITING ON LOADTEST
• bug 557966 - Add capabilities API entrypoint - DEFERRED TO SAGRADA
• bug 674904 - Script to generate project framework - DEFERRED TO SAGRADA

Configuration [GREEN]

The general structure of configuration is good, but there appear to be two approaches (generally matching .ini and .cfg suffixes) that have different semantics, and some of the differences are being compensated for outside of the config libraries. We need to figure out the source of differences, whether it is necessary, and encapsuate everything into the config libraries such that app.config = Config(filename) gives us a standard object to work with (one that largely shares the current hierarchy semantics, which are good).

Bugs:

• bug 675349 - Enforce [0-9A-Za-z_] as legal characters in section header names - DONE (slight changes)
• bug 675353 - Migrate GlobalConfFile wiki page to docs.services - DONE

Dynamic Loading [GREEN]

We currently have two approaches for dynamically generating interfaces - one for classes that implement a defined abstract interface, and one for more generic class loading. Looking generally good as is.

Bugs:

• bug 674777 - Add interface support to load_and_configure - DONE

Errors [GREEN]

It should be easy to find out whether a relevant exception has been defined in the core library.

Bugs:

• bug 675092 - Merge errors and response codes into a single exceptions file - DONE
• bug 673144 - Add more information about the source of the error into BackendError - DONE
• Add better error responses throughout all the codebases - DONE (though ongoing, too)

User Management [GREEN]

This has been recently rewritten to support a user-information-caching object to cut down on the number of queries into LDAP. Node assignment and reset codes have been spun off into their own library.

Bugs:

• bug 659156 - Change ldap to support getting new userids from there - DONE
• bug 675100 - Rewrite reg to support the changes to user management - DONE

Auth [YELLOW]

Technically, this is part of user management. However, it's worth calling out separately, because it is the user management piece that all our products tend to use.

Auth eventually needs to encapsulate a permissions model. This can be done through a token based permissions system that is generated once logged in. We'll use whatever the favored identification system to come out of browserID.

Bugs:

• bug 666237 - Cache authentication credentials for a period of time - DONE (pending secreview)

Test Support [GREEN]

We have the start of a test framework library, but it isn't yet to the point where it's easy to fire up a test and have access to the app. What utility functions would help in here?

Bugs:

• bug 675103 - Get rid of TestEnv class and make a saner way to do test initiation - DONE
• bug 675106 - Pull a lot of the replicated support functions into central tests/support.py - DONE
• bug 675376 - Make logging easier when running tests - DONE

Utility Functions [GREEN]

There are 30+ functions (and growing) in util.py. Time to curate!

Bugs:

• bug 675335 - Metabug for cleanup - ALL DONE
• bug 675334 - Create services.responses - DONE
• bug 675336 - Create services.crypto - NOT DOING
• bug 675337 - Create services.proxy - DONE
• bug 675339 - Create services.mail - DONE
• bug 675341 - Move extract_username and extract_node into services.user - DONE
• bug 675344 - Delete generate_reset_code and check_reset_code - DONE
• bug 675347 - Delete filter_params and convert_config - DONE

Reset Codes [RED]

This is newly separated from User Management so that it can be used anywhere else it is needed.

Bugs:

• bug 675348 - Pull reset codes out into their own package - DEFERRED TO Q4

Node Assignment [GREEN]

This is new service that has not yet been deployed. Once it has, zeus traffic to the User API call will be redirected to this service

Bugs:

• bug 659157 - Package and deploy new node-assignment - DONE

Logging [YELLOW]

What do we need to have a robust logging interface that doesn't get bogged down in complexity?

Bugs:

• Add support for metrics logging - DEFERRED TO SAGRADA

Auxiliary Support

Packaging [GREEN]

We need to make it as clean and easy to deploy packages as possible. The new makefiles have been a big help there. Much of the work can be done by the framework setup script, and one goal should be to minimize the number of places that need changing on an update.

Bugs (tarek):

• bug 672655 - Set everything up so that a version bump only needs to be done in one location - DONE
• Set everything up so that dependencies only need to be in one location - NOT DOING
• Pin server-full to production version

Loadtesting [YELLOW]

Load testing is being done using Jenkins and Funkload

Bugs (tarek):

• Have central loadtest scripts available for all products - INCOMPLETE (but vague)

Abuse Management

The Big Lebowski's scope has been drastically reduced. It is now spec'd as a RESTful API to manage IP white and black lists with TTLs. The API will accept commands from a single client: scripts run via InfraSec's monitoring appliance. Their appliance will manage all threshold violation detection and issue commands to the Big Lebowski which will manage the list of banned IPs on the Zeus instance(s).

Bugs (ryan):

Python Clients [RED]

There are bits and pieces of python sync clients littered throughout our repos. We should try to unify them into a complete, easy-to-use client interface that implements the entire* API.

(* I think we can ignore some of the deprecated options!)

Bugs (tarek/ryan):

• get crypto support for sync encryption/decryption into services.crypto - NOT DONE
• move delete and quota functions from account-portal to python-clients - NOT DONE
• work with hanson and stefan to see what's built and fill in the gaps. - NOT DONE

Jenkins [GREEN]

Jenkins is the automated unit-testing tool.

Bugs (tarek):

• Add documentation to docs.services release process that requires Jenkins to be green before a launch - DONE
• Fix up false-positives in Jenkins - DONE

Metrics [YELLOW]

Bugs (rob):

• Define a structure for metrics notifications - DONE (SAGRADA)
• Write a parser to consume them over a network stream (generated by the logger component) - MOVED TO SAGRADA

Mercurial [GREEN]

Bugs (toby):

• Add a services/deprecated folder - DONE (kind of)
• Clean up the services hg repo directory. - DONE