Services/ServerAppGuidelines
From MozillaWiki
< Services
- Server Environments
- Sandbox
- No puppet, just do stuff
- Internal only
- Dev
- Uses puppet
- Builds on checkin
- Runs functional + unit tests on checkin
- Devs have access to debug
- Same config as stage (and prod, if possible)
- Stage
- Uses puppet
- Requires an explicit update (see release process)
- No app gets into stage without full functional and stress test coverage
- Production
- Uses puppet
- Explicit push as per process below
- Functional tests run periodically in prodl
- Sandbox
- Release cycles
- Emergency pushes
- Must hit stage and pass functional tests. An unplanned outage (503 at Zeus level) is of higher value than pushing unverified code to prod.
- Requires sign-off from key people (TBD) before we short-circuit process
- QA signoff preferred, not mandated unless deemed necessary as a part of sign-off
- Normal maintenance
- Checkpoint on Wed AM, identify apps with pending updates that are stable + passing tests
- Stage by EOD Wed, hand off to QA Thursday AM, start stress (95% load) test
- QA signoff by EOD Monday (five days, three working days)
- Push to prod on Tuesday
- Debrief as needed at Wed meeting
- If we fail on stress/QA tests, we punt to the next week cycle
- Feature releases
- Checkpoint on Wed AM
- Ensure all supporting documentation for new/changed features is updated and in primary doc location
- Push to stage by EOD Wed
- QA + stress test as with normal releases, but for an extra week (12 days, eight working days) to ensure time to test new feature fully + extended run time to shake out issues
- Emergency pushes
- Documentation
- All new features and functional changes must be documented before the change can be pushed to stage
- Format is TBD, but current front-runner is Sphinx
- Build and Test
- Packaging
- All apps must use the same package and install mechanism, with services-core as a base to provide consistent behaviour
- Build scripts must follow a format that can be used with Hudson
- Continuous Integration
- All projects must be buildable by Hudson, and will build on each commit
- Build/test failures must be fixed immediately or the change reverted
- Functional and unit tests
- Tests are required for all apps as a prerequisite for pushing to stage
- Apps that are already in stage/prod require tests for all changes
- All features must be tested fully (not just default paths)
- Tests must run on each checkin, any test failures block push to stage
- Packaging