Requirements

PuppetAgain needs to be able to repeatably produce identical machines over long time ranges (years). That means carefully controlling which versions of which packages are available to be installed, both during machine creation and in runs of the puppet agent. However, it's impractical and unnecessary to specify the exact version of *every* package installed on a host.

The system needs to be able to manage periodic updates to packages, both for security fixes and normal software upgrades. This must be done in a controlled fashion, so that any effects on system behavior can be identified and handled appropriately.

We often need updated versions of distro-supplied packages, custom packages for existing applications, or packaged versions of internally-developed tools.

How all of this happens differs widely among supported operating systems!

Principles

• Do not allow the copying of a package to a repository to be the event that deploys a change -- that's un-trackable and difficult to revert
• Repositories form a strong isolation boundary: if you modify repository A, it's easy to guarantee that repository B hasn't changed. Repositories are cheap, so use lots of them.
• A repository's update cadence is part of its definition, so define that up front.
• Repositories should be for use by puppet only. Other consumers, such as mock environments, docker, etc., will have a different required update cadence and are unlikely to implement the proper resiliency to mirror failure, and thus should use repositories hosted elsewhere.

CentOS

CentOS: Repositories

Every puppetmaster hosts a bunch of yum repositories:

Notes:

• Repos under repos/yum/custom are custom-built for some purpose, and are not mirrored from anywhere. They may contain custom packages, or just packages culled from some mirror. Look for references to the repos in the puppet manifests to find their purpose.
• We generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind.
• The repos/yum/releng repos are deprecated; if you're tempted to add a package there, please create a new custom repository or update another existing custom repository instead
• The Fedora repos are unused and will be deleted soon
• Dynamic repositories are snapshots that are made on demand, where latest always points to the latest active snapshot. They are *not* automatically updated. The date on which they were most recently mirrored is given above.
• The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
• The CentOS base repos each have an images/ subdirectory, so they can be used with the 'url' command in a kickstart script

Mirror Synchronization Commands

Note: Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See bug 836014. The process takes about 10 minutes.

Note: From inside Mozilla, you'll need to use the Squid proxy: set RSYNC_PROXY to proxy.dmz.scl3.mozilla.com:3128

Remember that once changes land on the puppet master, they can be used for any purpose that needs packages. That includes kickstart, which always takes the most recent version of a package it can find! It also includes installs by puppet that specify version => "latest", and installs to fulfill requirements of packages defined by puppet. Think twice!

EPEL

rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/x86_64/ /data/repos/yum/mirrors/epel/6/2012-03-07/x86_64/
rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/i386/ /data/repos/yum/mirrors/epel/6/2012-03-07/i386/
time hardlink -v /data/repos/yum/mirrors/epel

Note that, as for CentOS below, the i386 mirrors were done much later than 2012-03-07 (July 2012).

CentOS 6

• Base and Updates (6.3 and higher)

CENTOS_FULL=6.5
rsync -v -n -aP    --filter='-r centos-version.txt'   --exclude isos --exclude drpms --exclude centosplus --exclude xen4 --exclude fasttrack --exclude contrib --exclude cr \
    --delete --delete-excluded rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_FULL/
time hardlink -v /data/repos/yum/mirrors/centos

Note that this pulls along 'SCL' and 'extras' and some other stuff. As long as it's not huge, it doesn't hurt.

Fedora 16

rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/fedora/linux/releases/16/Everything/i386/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/i386/os/
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://fedora.osuosl.org/fedora/linux/releases/16/Everything/x86_64/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/x86_64/os/
rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/i386/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/i386/
rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/x86_64/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/x86_64/
time hardlink -v /data/repos/yum/mirrors/fedora/

PuppetLabs

rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64/
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/products/i386/
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/dependencies/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64/
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/dependencies/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/dependencies/i386/
time hardlink -v /data/repos/yum/mirrors/puppetlabs/

Passenger

rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/x86_64/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/x86_64/
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/
time hardlink -v /data/repos/yum/mirrors/passenger/

node.js

# sync nodejs 6.10.0 packages
mkdir -p /data/repos/yum/mirrors/nodesource/el/6/x86_64 && cd $_
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-debuginfo-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-devel-6.10.0-1nodesource.el6.x86_64.rpm
curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-docs-6.10.0-1nodesource.el6.noarch.rpm
createrepo /data/repos/yum/mirrors/nodesource/el/6/x86_64/
time hardlink -v /data/repos/yum/mirrors/nodesource/

# sync C++11 packages (required by some nodejs libs)
mkdir -p /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS && cd $_
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-c++-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-libstdc++-devel-4.8.2-15.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-runtime-2.1-4.el6.noarch.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/scl-utils-20120927-8.el6.x86_64.rpm
curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-binutils-2.23.52.0.1-10.el6.x86_64.rpm
createrepo /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS/
time hardlink -v /data/repos/yum/mirrors/devtools-2/

HP

not everything is mirrored, so this just gets the good stuff

rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/x86_64/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/
rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/i386/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/
time hardlink -v /data/repos/yum/mirrors/hp/

CentOS: Installing Packages with Puppet

CentOS Packages are simple to install:

       CentOS: {
           package {
               "mypackage":
                   ensure => "1.2.3";
           }
       }

Generally, if it's important enough to install explicitly, it's important enough to pin a particular version. If you also need to pin versions for requirements, be sure you model the requirements with requires in puppet, so that puppet knows to install the requirements first.

CentOS: Building Custom Packages

See ReleaseEngineering/PuppetAgain/HowTo/Build RPMs for building RPMs.

CentOS: Adding New Packages

In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (yum install http://wherever.it.is/package.rpm) on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories. If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems. Upgrading librsync may be OK, but upgrading glibc or libopenssl this way might lead to a world of pain and sadness (noting that security releases often don't!). In the event that you need to add custom-built packages, you should know from the building process what the requirements are; otherwise it's pretty much the same process.

Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run createrepo (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.

Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose. For example, if you're updating openssh, create a new repos/yum/custom/openssh. If the appropriate repo already exists, use it. Try to include SRPMs as well! When setting up a custom repo, it's helpful to include an `update.sh` script in the root of the repo that can be used to re-mirror it later. Check other update scripts for useful techniques (for example, `repotrack` is pretty useful!)

Once you've assembled a directory containing the proper packages, run

 createrepo --update .

in that directory to update the metadata. Don't forget to run puppetmaster-fixperms afterward to make sure permissions are correct.

If you added a new repository, you'll need to refer to it from the puppet configs. Add a clause to modules/packages/manifests/setup.pp, either a regular packages::yumrepo if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with @) if it should only be available on some hosts. Only installing the repo on some hosts limits the carnage if something goes wrong with the repo.

Then, write or update the classes under modules/packages. If your repository is virtual, you'll need to add something like realize(Packages::Yumrepo['passenger']) to the package class to ensure the repo is in place.

Note: Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream

If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp:

  # to flush the package index, increase this value by one (or
  # anything, really, just change it). 
- $repoflag = 5
+ $repoflag = 6

This will cause all CentOS machines to run yum cache clean all.

Ubuntu

Reference Links

• https://wiki.debian.org/RepositoryFormat
• debpartial-mirror README

Ubuntu: Repositories

Notes:

• Dynamic repositories are snapshots that are made on demand, where latest always points to the latest active snapshot. They are *not* automatically updated. The date on which they were most recently mirrored is given above.
• This arrangement can't be updated (see the warning below), so we will likely need to change to something more like the CentOS repositories.

Mirror Synchronization Commands

Warning: The debmirror command includes the equivalent of rsync's --delete. If you try to mirror only one section, dist, arch, etc., debmirror will delete the section/dist/arch you omit. Don't do that. See bug 844172.

Setup

GNUPGHOME has the Ubuntu arch key in it. If a sync operation fails because a signature does not verify, download the key using

GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg \
  --keyserver keyserver.ubuntu.com --recv-keys $KEY_ID

Add `--keyserver-options http-proxy=proxy.dmz.scl3.mozilla.com:3128` at Mozilla. Note that this keyserver's search option appears to be broken. You can usually google for the key id, and find the relevant link on the keyserver, and then copy-paste the result into

GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import

with `/etc/debmirror.conf` containing only the Perl no-op "1;".

Note that `hardlink` isn't used here - debmirror's name-based deduplication squeezes out just about all the space possible.

Ubuntu

We mirror all Ubuntu releases to the same directory, using --nocleanup to prevent deletion of packages not touched in the current mirror operation. Note that we mirror 'universe', too. Although it's huge, puppet relies on some packages in that section.

For example, the following mirrors precise. The DIST can be changed to e.g., mirror a different version, or just mirror security.

SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer
DIST=precise,precise-security
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
   -a $ARCH -s $SECTION -d $DIST \
   -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \
   --nocleanup \
   --dry-run \
   /data/repos/apt/ubuntu/

xorg-edgers

SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer
DIST=precise
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
   -a $ARCH -s $SECTION -d $DIST \
   -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \
   --dry-run \
   --nocleanup \
   /data/repos/apt/xorg-edgers/

nginx-development

SECTION=main
DIST=precise,trusty
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
    -a $ARCH -s $SECTION -d $DIST \
    -h ppa.launchpad.net -r /nginx/development/ubuntu --rsync-extra=none -e http --progress \
    --dry-run \
    --nocleanup \
    /data/repos/apt/nginx-development/

puppetlabs

 wget https://apt.puppetlabs.com/pubkey.gpg
 GNUPGHOME=/etc/debmirror-gpg  gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import pubkey.gpg

 SECTION=main,dependencies
 DIST=precise,trusty
 ARCH=i386,amd64
 GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
    -a $ARCH -s $SECTION -d $DIST \
    -h apt.puppetlabs.com -r / -e http --progress \
    --dry-run \
    /data/repos/apt/puppetlabs/

Ubuntu: Installing Packages with Puppet

Simple:

       Ubuntu: {
           package {
               "mypackage":
                   ensure => "1.2.3";
           }
       }

As with CentOS packages, it's generally a good idea to pin the version of things that are important enough to be named in puppet. If requirements need to be pinned too, then model the requirements relationship properly with requires.

Ubuntu: Building Custom Packages

See ReleaseEngineering/PuppetAgain/HowTo/Build DEBs for details on building DEBs.

Ubuntu: Adding New Packages

Warning: This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the custom repositories) and packages installed by puppet (which does).

Sometimes you may need to update only one package from upstream without syncing the whole repo. Or you may have custom-built packages that you need to install. In either case, the tool for the job is a new, custom, repository.

Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run createrepo (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.

Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose. For example, if you're updating openssh, create a new repos/apt/custom/openssh. If the appropriate repo already exists, use it.

To build a custom repository, start by laying out your package pool. The best plan is to divide the packages by dist. For example:

[root@releng-puppet2.srv.releng.scl3.mozilla.com openssl]# find pool/
pool/
pool/trusty
pool/trusty/openssl_1.0.1f-1ubuntu2.5_i386.deb
pool/trusty/openssl_1.0.1.orig.tar.gz
pool/trusty/libssl1.0.0-dbg_1.0.1f-1ubuntu2.5_i386.deb
...
pool/precise
pool/precise/libssl1.0.0_1.0.1-4ubuntu5.17_i386.deb
...

Then edit an update.sh in the root of the repository with something along the lines of

for arch in i386 amd64; do
  for dist in precise trusty; do
    mkdir -p dists/${dist}/all/binary-$arch
    dpkg-scanpackages --multiversion --arch $arch pool/$dist > dists/${dist}/all/binary-$arch/Packages
    bzip2 < dists/${dist}/all/binary-$arch/Packages > dists/${dist}/all/binary-$arch/Packages.bz2
  done
done

This script will update the apt indexes. Note that the `--multiversion` is required if the repo is to contain multiple versions of the same package, like yum repositories can.

Don't forget to run puppetmaster-fixperms afterward to make sure permissions are correct.

If you added a new repository, you'll need to refer to it from the puppet configs. Add a clause to modules/packages/manifests/setup.pp, either a regular packages::aptrepo if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with @) if it should only be available on some hosts. Only installing the repo on some hosts limits the carnage if something goes wrong with the repo.

Then, write or update the classes under modules/packages. If your repository is virtual, you'll need to add something like realize(Packages::Aptrepo['xorg-edgers']) to the package class to ensure the repo is in place.

If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp:

  # to flush the package index, increase this value by one (or
  # anything, really, just change it). 
- $repoflag = 5
+ $repoflag = 6

This will cause all Ubuntu machines to run apt-get update.

Automatically Pulling Dependencies

Note: This is not at all clear, sorry -- there are no real experts on this topic at Mozilla, so learn what you can and update the wiki!

This needs to be done on an Ubuntu machine. You need to use debpartial-mirror, apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes:

# http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates.sh
debpartial-mirror -c ./releng-updates.conf all
rm -rf releng-updates/dists
cd releng-updates
for arch in i386 amd64; do
  mkdir -p dists/precise-updates/all/binary-$arch
  apt-ftparchive --arch $arch packages . > dists/precise-updates/all/binary-$arch/Packages
  bzip2 < dists/precise-updates/all/binary-$arch/Packages > dists/precise-updates/all/binary-$arch/Packages.bz2
done

releng-updates.conf

[GLOBAL]
;debug = DEBUG
mirror_dir = ./
architectures = i386 amd64
components = main
distributions = precise

get_suggests = true
get_recommends = true
get_provides = true
get_sources = true
get_packages = true

[releng-updates]
server = http://archive.ubuntu.com/ubuntu
distributions = precise-updates
components = main restricted universe
# specify needed packages here
filter = name:gnome-settings-daemon

Darwin

Darwin: Repositories

There's no such thing as a repository for OS X packages, sadly. DMGs are stored in repos/DMGs. DMGs are generally built for a specific OS version and put in per-os-version subdirectories, although DMGs that are compatible across versions are in the root. Each DMG is named $packagename-$version.dmg.

Darwin: Installing Packages with Puppet

Use the packages::pkgdmg defined type to install DMGs, giving the package name as the resource name and the package version in the version parameter. The type will construct the correct filename from this information. For example:

       Darwin: {
           Anchor['packages::wget::begin'] ->
           packages::pkgdmg {
               wget:
                   version => "1.12-1";
           } -> Anchor['packages::wget::end']
       } 

DMGs that are downloaded as-is should be documented briefly (at least with the source) in the manifest files that install them. For example (from modules/packages/manifests/puppet.pp):

       Darwin: {
           # These DMGs come directly from http://downloads.puppetlabs.com/mac/
           Anchor['packages::puppet::begin'] ->
           packages::pkgdmg {
               puppet:
                   version => $puppet_version;
               facter:
                   version => $facter_version;
           } -> Anchor['packages::puppet::end']
       }

For DMGs that are *not* os-version-specific, pass os_version_specific => false.

Darwin: Custom Packages

DMGs that are custom built should have a shell script in modules/packages/manifests named $package-dmg.sh which builds the DMG. If there is a corresponding RPM (custom or stock) for the package, then the shell script can require that the source RPM be unpacked first. See ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs for more details.