Labs/Bespin/ProductionDeployment
Contents
Production Deployment
This document describes the production deployment of the Python backend. The Python backend is a standard WSGI application, which means that there are numerous ways to deploy it. There are even standalone WSGI servers that you can deploy it with, including one that comes with Paste and Spawning.
Building
Bespin uses Paver to handle its builds. The top-level pavement file knows how to package up Bespin for a production deployment. At Bespin's top-level directory, run the following command to build Bespin:
paver dist
This command will compress the JavaScript, put together a Python package and produce a complete tarball that is ready to be untarred on the production server. The final tarball lives in build/BespinServer.tar.gz.
Installing
Copy the tarball built in the previous step to your production server. It will go into the home directory of the account that you'll be running the app as. We use mod_wsgi's daemon model (vs. in-process model), which allows us to run Bespin as a different user than the web server.
untar the file, and you will end up with a BespinServer directory. In there, run:
python bootstrap.py --no-site-packages
This will set up a virtualenv and then install all of the dependencies. This installation procedure requires that the production server have outbound access to the internet to grab the packages required by Bespin. If the production server does not have outbound access, you can look into packaging up the dependencies into a single collection. We're using pip to install the dependencies, and there are some options to help collect up the requirements.
- as of 5/20/09 2 packages path and Paste had conflicts with pip install. Manually install path and install it:
bin/pip install http://pypi.python.org/packages/source/p/path.py/path-2.2.zip
the latest release of Paste didn't include the required patches for bespin. Manually copy over Paste from your dev system to production system under <mybespindirectory>/libs and install:
bin/pip install libs/Paste-1.7.3dev-r7791.tar.gz
Apache with mod_wsgi
We chose Apache with mod_wsgi because that combination will automatically manage the Python processes for us. We are using mod_wsgi 2.3. It is not uncommon for people to Apache or nginx with a proxy to a Python web server.
Here is the Bespin part of our Apache configuration with comments inline to describe the different parts of the config:
# gzip all of our plaintext resources to make downloads faster AddOutputFilterByType DEFLATE text/html text/plain text/xml text/javascript application/x-javascript # give Apache access to the directory where we will store the Bespin WSGI script <Directory /home/WSGIUSER/wsgi-apps> Order allow,deny Allow from all </Directory> # Optional, give Apache access to the static files so that it can serve them directly <Directory /home/WSGIUSER/BespinServer/frontend> Order allow,deny Allow from all </Directory> # our production setup includes a caching load balancer in front. # we tell the load balancer to cache for 5 minutes. # we can use the commented out lines to tell it to not cache, # which is useful if we're updating. <FilesMatch "\.(html|js|png|jpg|css)$"> ExpiresDefault "access plus 5 minutes" ExpiresActive On #Header unset cache-control: #Header append cache-control: "no-cache, must-revalidate" </FilesMatch> # tell Apache where to find our index file. We sometimes will put up # a temporary message while doing maintenance #AliasMatch ^/$ /home/WSGIUSER/BespinServer/frontend/temporary.html AliasMatch ^/$ /home/WSGIUSER/BespinServer/frontend/index.html # here's the mod_wsgi configuration. We're setting up 2 processes, each # with 15 threads. Note that the root of the server is served by # bespin.wsgi. The WSGIPythonHome points to a vitualenv directory # into which all of the libraries are installed. WSGIScriptAlias / /home/WSGIUSER/wsgi-apps/bespin.wsgi WSGIDaemonProcess bespin user=WSGIUSER processes=2 threads=15 WSGIProcessGroup bespin WSGIPythonHome /home/WSGIUSER/BespinServer WSGISocketPrefix run/wsgi # serve up the static files directly from Apache. # The python web server *can* serve the static files, but Apache # is much more efficient at it. Alias /images/ /home/WSGIUSER/BespinServer/frontend/images/ Alias /css/ /home/WSGIUSER/BespinServer/frontend/css/ Alias /js/ /home/WSGIUSER/BespinServer/frontend/js/ #Alias /index.html /home/WSGIUSER/BespinServer/frontend/temporary.html #Alias /editor.html /home/WSGIUSER/BespinServer/frontend/temporary.html #Alias /dashboard.html /home/WSGIUSER/BespinServer/frontend/temporary.html Alias /index.html /home/WSGIUSER/BespinServer/frontend/index.html Alias /editor.html /home/WSGIUSER/BespinServer/frontend/editor.html Alias /dashboard.html /home/WSGIUSER/BespinServer/frontend/dashboard.html Alias /docs/code/ /home/WSGIUSER/BespinServer/frontend/js/ Alias /docs/ /home/WSGIUSER/BespinServer/docs/
Most of the configuration above is optional. In fact, the only parts required are:
- directory directive to provide access to the wsgi-apps directory (where you put the bespin.wsgi script)
- The WSGI* directives that tell mod_wsgi what to run and how to run it.
The WSGI script
mod_wsgi scripts are standard Python scripts that include an "application" object in their namespace. That is the WSGI application that is attached to the server.
Bespin's run time configuration is all stored in variables in the bespin.config module in a dictionary-like object called "c". The WSGI script *could* do all of the itself by directly setting the values. In the script below, we set the values via a simple .ini file. Additionally, we set up Python's logging system to put any log output that interests us into logs/bespin.log.
import os
import logging
import logging.handlers
import ConfigParser
from bespin import config, controllers
import sys
#redirect stdout to go to stderr, it will appear in the apache error log
sys.stdout = sys.stderr
# setting the configuration profile starts us off with reasonable values for our
# environment.
config.set_profile("prod")
# load config values from the INI file
cp = ConfigParser.ConfigParser()
cp.read(os.path.abspath(os.path.dirname(__file__) + "/../config/bespin.ini"))
# config values are available as a dictionary, and config.c is a dictionary
# so we can just do config.c.update.
config.c.update(cp.items("config"))
# set up logging
root_log = logging.getLogger()
root_log.setLevel(logging.WARN)
handler = logging.handlers.RotatingFileHandler(
os.path.abspath(os.path.dirname(__file__) + "/../logs/bespin.log"))
root_log.addHandler(handler)
handler.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
# email out errors
handler2 = logging.handlers.SMTPHandler('localhost', 'YOU@YOU.COM',
['YOU@YOU.COM'],
'Bespin Error')
handler2.setFormatter(logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s"))
handler2.setLevel(logging.ERROR)
# uncomment the next line if you actually want the errors mailed, the
# above is sample config
#root_log.addHandler(handler2)
# activate profile is responsible for taking the config and creating database
# connections, etc.
config.activate_profile()
# this is the bit for mod_wsgi. controllers.make_app() gives us the top-level WSGI
# application.
application = controllers.make_app()
If you don't plan to run a full asynchronous setup on the server (requiring you to install beanstalkd and run bespin_worker processes), you'll want to set this flag *before* the config.activate_profile() in the WSGI script.
config.c.async_jobs=False
The .ini file
The INI file is where your site-specific configuration will go.
[config] dburl=mysql://user:password@localhost/bespin secret=YOUR SECRET USED IN HASHING static_dir=/home/WSGIUSER/BespinServer/frontend fsroot=YOUR HOME DIRECTORY FOR ALL BESPIN FILES
Creating the database
Assuming bespin.wsgi is in a directory called wsgi-apps that is at the same level as your BespinServer directory, you can then run
paver create_db
To set up your production database.
