Web Operations/Best Practices

Best Practices

This space is intended to serve as a reference for anyone who is working on a project which the WebOps team will deploy and support. This is intended to be a collaborative page that can be added to by anyone with advice around designing applications which will run on WebOps supported infrastructure.

As a developer you should...

• Not hard code localhost anywhere
• Use IT supported services whenever possible (see service matrix)
• Do not assume every HTTP request will be serviced by the same machine
  • Don't use local file storage
    • Instead: use a database, NFS, S3, Swift, etc
• Assume you cannot directly install packages from pip/npm
• Elasticsearch
  • ES indexes should be abstracted through the use of aliases[1]. This allows you to search multiple indexes, reindex data with no downtime[2], or seamlessly rotate indexes (dropping data that is not as useful anymore). Reindexing is particularly useful as it defragments indexes, making them take up less disk space and may improve the speed of search operations.
  • Be prepared to really test what kind of shard allocation strategy you want to use. Ideally, shards should roughly the same size, with enough shards to properly distribute the work but not so many shards that the overhead of maintaining them decreases search efficiency.
  • Your application should able to cope with ES downtime or delays. Depending on the nature of your application, this could mean throttling requests or queuing them up if the ES cluster becomes slow or unresponsive.
  • Think about how to manage index failure. If at all possible, ES should not be your primary data store. ES is less resilient to node failure that widely advertised, leading to data loss[3]. While there are more options than there used to be for backing up ES data, it is possible for acknowledged writes to be lost. Depending on what your application is doing, you may want to consider storing a "window" of primary documents that can be used to restore a "corrupted" ES index.
• Do not purchase domain names. When domains are purchased by people (not Mozilla Corp) it takes substantial time and effort to transfer them to Mozilla ownership. This leads to lengthy delays during deployment, especially if you need SSL support. Instead file a bug for WebOps and mark it as confidential. Marking it confidential will prevent third parties from registering it and ransoming it back to Mozilla.
• If you deploy your application using the following layout, you will get things like backups and monitoring for free.
  • Place application files in bla bla bla, etc…
• For some tips on Application Caching for off-line use, check out these articles:
  • http://alistapart.com/article/offline-first
  • http://alistapart.com/article/application-cache-is-a-douchebag