Web Operations/Reference Specification/Discussion Pages/puppet

Overview

This is a discussion around the current state of infra puppet and some proposed solutions.

This is prefaced with the notion that this is currently only from the perspective of WebOps and may not hold true for all of IT. This is the case for both the current state and the proposed solutions.

Current State

The current state of puppet is a bit unmaintainable. We do not use top level or standard modules for deploying. For example, there is a custom top level Apache module but it is almost unused when deploying webaps. Webaps are generally deployed utilizing a custom Apache class which is built into the webaps module. The same can be said for deploying memcache or reddis and other similar technologies. This presents problems due to maintainability as well as best practice.

There is little use of templates for deploying applications. As we use puppet today, most configurations are stored in flat files per application. Worse still these configuration flat files are duplicated for each deployment of an application (dev / stage / prod). One of the key advantages to using a configuration management system like Puppet is the reduction in operator errors (typos etc) that come from using templates.

There is no development system for testing changes to puppet. This is no small matter and has, for example taken down large web properties due to a small misconfiguration. When using a configuration management system configuration code is often shared between multiple deployments. This is by design, however it presents challenges when changes are introduced into the system. It becomes almost impossible to make changes while assuring uptime without the ability to test these changes in a development environment.

We store application specific configuration in puppet. This nearly eliminates the possibility of creating a self-service platform as it requires our consumers to go through us (file bugs) to update application settings. These settings range from API keys to database settings. This slows down the time to market as it blocks developers on our availability. Additionally it increases the chances for a typo or leak of sensitive information (ie: database password) due to the human exchange of this information.

Proposed Changes

We are proposing that wherever possible we only use existing open source puppet modules. For most of the technologies we deploy there are excellent and comprehensive modules that are open source and have been produced by either Puppet Labs or the technology vendor themselves. These modules are generally far more comprehensive than the modules we are using today. Furthermore these modules tend to be highly tested by a large community which would reduce the changes of our discovering new bugs.

We are further proposing a system where the entire puppet tree will be contained in git. This allows us to have easy to use development branches, either in the main repository or by forking. This also would allow us to introduce a pier-review process for critical changes through the existing pull-request module.

Once the entire puppet tree exists in git and is using community modules we are in a position where it is not only easy, but a near necessity to use templates for everything. Most modules ship with templates already which largely solves this issue outright.

We propose that no application level data will be stored in puppet. This data should either ship with the application itself (local configs or deployment YAML file) or be autogenerated and available via environment variables (ie: database credentials) or be stored in a key-value store. We are looking at using Heira backed by Zookeeper for this key-value store. This way the valuse can be consumed by puppet through hiera or by the applications through the zookeeper API.

Puppet Layout

The new puppet layout, while still in design and subject to alteration, consists of a Back End Module (BEM), a Front End Module (FEM), an application YAML file and documentation.

The BEM is the module that we get from the community. These modules *should* not require any modification by us. In point of fact, if we are considering modifying these modules we should instead analyze our deployment strategy and think about modifying that instead.

The FEM is stored in our git repository and is where we set sane defaults for our environment. It is in the FEM that we can introduce any adjustments necessary to align with our deployment methodology, however we caution against major modification here as well, as this will lead to increased difficulty upgrading the underlying BEMs.

The YAML file is controlled by the consumer and will generally live in their code repository. In other words it lives with the web application or the deployment repository for a system. This file will contain only minor amounts of puppet code, in some cases none, and basically just instantiates our FEMs with any parameter overrides necessary for the specific project, that is anything contrary to the defaults we define in the FEM.

We will provide documentation, including examples for each and every FEM. This is absolutely critical for this platform to be usable in a self-service manor. This documentation should link to the BEMs documentation and/or git repository. Additionally it will include links to the FEM unless it is the README for said module.