ReleaseEngineering/PuppetAgain/Packages
Contents
Requirements
PuppetAgain needs to be able to repeatably produce identical machines over long time ranges (years). That means carefully controlling which versions of which packages are available to be installed, both during machine creation and in runs of the puppet agent. However, it's impractical and unnecessary to specify the exact version of *every* package installed on a host.
The system needs to be able to manage periodic updates to packages, both for security fixes and normal software upgrades. This must be done in a controlled fashion, so that any effects on system behavior can be identified and handled appropriately.
We often need updated versions of distro-supplied packages, custom packages for existing applications, or packaged versions of internally-developed tools.
How all of this happens differs widely among supported operating systems!
Principles
- Do not allow the copying of a package to a repository to be the event that deploys a change -- that's un-trackable and difficult to revert
- Repositories form a strong isolation boundary: if you modify repository A, it's easy to guarantee that repository B hasn't changed. Repositories are cheap, so use lots of them.
- A repository's update cadence is part of its definition, so define that up front.
- Repositories should be for use by puppet only. Other consumers, such as mock environments, docker, etc., will have a different required update cadence and are unlikely to implement the proper resiliency to mirror failure, and thus should use repositories hosted elsewhere.
CentOS
CentOS: Repositories
Every puppetmaster hosts a bunch of yum repositories:
Notes:
- Repos under repos/yum/custom are custom-built for some purpose, and are not mirrored from anywhere. They may contain custom packages, or just packages culled from some mirror. Look for references to the repos in the puppet manifests to find their purpose.
- We generally try to mirror source RPMs for all repositories; this way, if we need to make a small fix to such an RPM, we can easily find the source for it without resorting to things like rpmfind.
- The repos/yum/releng repos are deprecated; if you're tempted to add a package there, please create a new custom repository or update another existing custom repository instead
- The Fedora repos are unused and will be deleted soon
- Dynamic repositories are snapshots that are made on demand, where latest always points to the latest active snapshot. They are *not* automatically updated. The date on which they were most recently mirrored is given above.
- The current CentOS version is available at http://repos/repos/yum/mirrors/centos/6/latest/centos-version.txt.
- The CentOS base repos each have an images/ subdirectory, so they can be used with the 'url' command in a kickstart script
Mirror Synchronization Commands
Note: Each of these includes a 'hardlinks' command which can find duplicate files and hard-link them together, saving a bit of space. See bug 836014. The process takes about 10 minutes.
Note: From inside Mozilla, you'll need to use the Squid proxy: set RSYNC_PROXY to proxy.dmz.scl3.mozilla.com:3128
Remember that once changes land on the puppet master, they can be used for any purpose that needs packages. That includes kickstart, which always takes the most recent version of a package it can find! It also includes installs by puppet that specify version => "latest", and installs to fulfill requirements of packages defined by puppet. Think twice!
EPEL
rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/x86_64/ /data/repos/yum/mirrors/epel/6/2012-03-07/x86_64/ rsync -n --no-p -rv --delete --size-only rsync://mirror.unl.edu/fedora-epel/6/i386/ /data/repos/yum/mirrors/epel/6/2012-03-07/i386/ time hardlink -v /data/repos/yum/mirrors/epel
Note that, as for CentOS below, the i386 mirrors were done much later than 2012-03-07 (July 2012).
CentOS 6
- Base and Updates (6.3 and higher)
CENTOS_FULL=6.5
rsync -v -n -aP --filter='-r centos-version.txt' --exclude isos --exclude drpms --exclude centosplus --exclude xen4 --exclude fasttrack --exclude contrib --exclude cr \
--delete --delete-excluded rsync://linux.mirrors.es.net/centos/$CENTOS_FULL/ /data/repos/yum/mirrors/centos/$CENTOS_FULL/
time hardlink -v /data/repos/yum/mirrors/centos
Note that this pulls along 'SCL' and 'extras' and some other stuff. As long as it's not huge, it doesn't hurt.
Fedora 16
rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://mirrors.usc.edu/fedora/linux/releases/16/Everything/i386/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/i386/os/ rsync -n --no-p --delete --size-only -rv --exclude=EFI --exclude=drpms --exclude=images --exclude=isolinux --exclude='RPM-GPG-KEY*' rsync://fedora.osuosl.org/fedora/linux/releases/16/Everything/x86_64/os/ /data/repos/yum/mirrors/fedora/16/2012-03-07/releases/Everything/x86_64/os/ rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/i386/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/i386/ rsync -n --no-p --delete --size-only -rv --exclude=drpms --exclude=debug rsync://fedora.osuosl.org/fedora/linux/updates/16/x86_64/ /data/repos/yum/mirrors/fedora/16/2012-03-07/updates/x86_64/ time hardlink -v /data/repos/yum/mirrors/fedora/
PuppetLabs
rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/products/x86_64/ rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/products/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/products/i386/ rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/dependencies/x86_64/ /data/repos/yum/mirrors/puppetlabs/el/6/dependencies/x86_64/ rsync -n -rLv rsync://yum.puppetlabs.com/packages/yum/el/6/dependencies/i386/ /data/repos/yum/mirrors/puppetlabs/el/6/dependencies/i386/ time hardlink -v /data/repos/yum/mirrors/puppetlabs/
Passenger
rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/x86_64/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/x86_64/ rsync -n -av rsync://passenger.stealthymonkeys.com/rpms/rhel/6/i386/ /data/repos/yum/mirrors/passenger/rhel/6/2012-07-05/i386/ time hardlink -v /data/repos/yum/mirrors/passenger/
node.js
# sync nodejs 6.10.0 packages mkdir -p /data/repos/yum/mirrors/nodesource/el/6/x86_64 && cd $_ curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-6.10.0-1nodesource.el6.x86_64.rpm curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-debuginfo-6.10.0-1nodesource.el6.x86_64.rpm curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-devel-6.10.0-1nodesource.el6.x86_64.rpm curl -O https://rpm.nodesource.com/pub_6.x/el/6/x86_64/nodejs-docs-6.10.0-1nodesource.el6.noarch.rpm createrepo /data/repos/yum/mirrors/nodesource/el/6/x86_64/ time hardlink -v /data/repos/yum/mirrors/nodesource/
# sync C++11 packages (required by some nodejs libs) mkdir -p /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS && cd $_ curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-4.8.2-15.el6.x86_64.rpm curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-gcc-c++-4.8.2-15.el6.x86_64.rpm curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-libstdc++-devel-4.8.2-15.el6.x86_64.rpm curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-runtime-2.1-4.el6.noarch.rpm curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/scl-utils-20120927-8.el6.x86_64.rpm curl -O https://people.centos.org/tru/devtools-2/6/x86_64/RPMS/devtoolset-2-binutils-2.23.52.0.1-10.el6.x86_64.rpm createrepo /data/repos/yum/mirrors/devtools-2/6/x86_64/RPMS/ time hardlink -v /data/repos/yum/mirrors/devtools-2/
HP
not everything is mirrored, so this just gets the good stuff
rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/x86_64/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/x86_64/current/ rsync -n -av downloads.linux.hp.com::SDR/psp/CentOS/6/i386/current/ /data/repos/yum/mirrors/hp/proliantsupportpack/CentOS/6/i386/current/ time hardlink -v /data/repos/yum/mirrors/hp/
CentOS: Installing Packages with Puppet
CentOS Packages are simple to install:
CentOS: {
package {
"mypackage":
ensure => "1.2.3";
}
}
Generally, if it's important enough to install explicitly, it's important enough to pin a particular version. If you also need to pin versions for requirements, be sure you model the requirements with requires in puppet, so that puppet knows to install the requirements first.
CentOS: Building Custom Packages
See ReleaseEngineering/PuppetAgain/HowTo/Build RPMs for building RPMs.
CentOS: Adding New Packages
In the event you find a need for an updated package from newer CentOS repositories, first try installing that package manually (yum install http://wherever.it.is/package.rpm) on a target host, to ensure that it doesn't have any requirements that aren't satisfied from the mirrored repositories. If there are any such requirements, consider carefully how many of them to cherry-pick out of the repository, and the effects that will have on other systems. Upgrading librsync may be OK, but upgrading glibc or libopenssl this way might lead to a world of pain and sadness (noting that security releases often don't!). In the event that you need to add custom-built packages, you should know from the building process what the requirements are; otherwise it's pretty much the same process.
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run createrepo (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.
Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose. For example, if you're updating openssh, create a new repos/yum/custom/openssh. If the appropriate repo already exists, use it. Try to include SRPMs as well! When setting up a custom repo, it's helpful to include an `update.sh` script in the root of the repo that can be used to re-mirror it later. Check other update scripts for useful techniques (for example, `repotrack` is pretty useful!)
Once you've assembled a directory containing the proper packages, run
createrepo --update .
in that directory to update the metadata. Don't forget to run puppetmaster-fixperms afterward to make sure permissions are correct.
If you added a new repository, you'll need to refer to it from the puppet configs. Add a clause to modules/packages/manifests/setup.pp, either a regular packages::yumrepo if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with @) if it should only be available on some hosts. Only installing the repo on some hosts limits the carnage if something goes wrong with the repo.
Then, write or update the classes under modules/packages. If your repository is virtual, you'll need to add something like realize(Packages::Yumrepo['passenger']) to the package class to ensure the repo is in place.
Note: Packages used in the mock environment require both i686 and x86_64 packages to be in the x86_64 repo, just like upstream
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp:
# to flush the package index, increase this value by one (or # anything, really, just change it). - $repoflag = 5 + $repoflag = 6
This will cause all CentOS machines to run yum cache clean all.
Ubuntu
Reference Links
Ubuntu: Repositories
| url | repository | arch | section | dist | mirror date |
|---|---|---|---|---|---|
| repos/apt/ubuntu | Ubuntu 12.04 and 14.04 | i386,amd64 | main,restricted,universe | precise,precise-security,trusty,trusty-security (note: no precise-updates) | varies |
| repos/apt/xorg-edgers | xorg-edgers fresh X Crack | i386,amd64 | main,restricted,universe | precise | 2013-02-21 |
| repos/apt/releng | custom-built packages | i386,amd64 | main,restricted,universe | precise | |
| repos/apt/releng-updates | partial mirror of precise-updates | i386,amd64 | main,restricted,universe | precise | |
| repos/apt/puppetlabs | mirror of apt.puppetlabs.com | i386,amd64 | main,restricted,universe | precise | |
| repos/apt/precise-updates | mirror of precise-updates | i386,amd64 | main,restricted,universe | precise-updates |
Notes:
- Dynamic repositories are snapshots that are made on demand, where latest always points to the latest active snapshot. They are *not* automatically updated. The date on which they were most recently mirrored is given above.
- This arrangement can't be updated (see the warning below), so we will likely need to change to something more like the CentOS repositories.
Mirror Synchronization Commands
Warning: The debmirror command includes the equivalent of rsync's --delete. If you try to mirror only one section, dist, arch, etc., debmirror will delete the section/dist/arch you omit. Don't do that. See bug 844172.Setup
GNUPGHOME has the Ubuntu arch key in it. If a sync operation fails because a signature does not verify, download the key using
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg \ --keyserver keyserver.ubuntu.com --recv-keys $KEY_ID
Add `--keyserver-options http-proxy=proxy.dmz.scl3.mozilla.com:3128` at Mozilla. Note that this keyserver's search option appears to be broken. You can usually google for the key id, and find the relevant link on the keyserver, and then copy-paste the result into
GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import
with `/etc/debmirror.conf` containing only the Perl no-op "1;".
Note that `hardlink` isn't used here - debmirror's name-based deduplication squeezes out just about all the space possible.
Ubuntu
We mirror all Ubuntu releases to the same directory, using --nocleanup to prevent deletion of packages not touched in the current mirror operation. Note that we mirror 'universe', too. Although it's huge, puppet relies on some packages in that section.
For example, the following mirrors precise. The DIST can be changed to e.g., mirror a different version, or just mirror security.
SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer DIST=precise,precise-security ARCH=i386,amd64 GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \ -a $ARCH -s $SECTION -d $DIST \ -h us.archive.ubuntu.com -r /ubuntu -e rsync --progress \ --nocleanup \ --dry-run \ /data/repos/apt/ubuntu/
xorg-edgers
SECTION=main,main/debian-installer,restricted,restricted/debian-installer,universe,universe/debian-installer DIST=precise ARCH=i386,amd64 GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \ -a $ARCH -s $SECTION -d $DIST \ -h ppa.launchpad.net -r /xorg-edgers/ppa/ubuntu --rsync-extra=none -e http --progress \ --dry-run \ --nocleanup \ /data/repos/apt/xorg-edgers/
nginx-development
SECTION=main
DIST=precise,trusty
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
-a $ARCH -s $SECTION -d $DIST \
-h ppa.launchpad.net -r /nginx/development/ubuntu --rsync-extra=none -e http --progress \
--dry-run \
--nocleanup \
/data/repos/apt/nginx-development/
puppetlabs
wget https://apt.puppetlabs.com/pubkey.gpg GNUPGHOME=/etc/debmirror-gpg gpg --no-default-keyring --keyring /etc/debmirror-gpg/trustedkeys.gpg --import pubkey.gpg
SECTION=main,dependencies
DIST=precise,trusty
ARCH=i386,amd64
GNUPGHOME=/etc/debmirror-gpg/ debmirror --config-file=/etc/debmirror.conf --source \
-a $ARCH -s $SECTION -d $DIST \
-h apt.puppetlabs.com -r / -e http --progress \
--dry-run \
/data/repos/apt/puppetlabs/
Ubuntu: Installing Packages with Puppet
Simple:
Ubuntu: {
package {
"mypackage":
ensure => "1.2.3";
}
}
As with CentOS packages, it's generally a good idea to pin the version of things that are important enough to be named in puppet. If requirements need to be pinned too, then model the requirements relationship properly with requires.
Ubuntu: Building Custom Packages
See ReleaseEngineering/PuppetAgain/HowTo/Build DEBs for details on building DEBs.
Ubuntu: Adding New Packages
Warning: This procedure can cause package version conflicts between packages installed during kickstart (which does not reference the custom repositories) and packages installed by puppet (which does).Sometimes you may need to update only one package from upstream without syncing the whole repo. Or you may have custom-built packages that you need to install. In either case, the tool for the job is a new, custom, repository.
Once you know the complete set of packages required, but before copying anything onto the puppet masters, "pin" the versions of the package in question and any requirements in puppet to what they are before your change, and deploy that patch. This provides a backout path for you later to install exactly the versions that were installed before your changes. Only when that change is deployed, add the new packages to the releng repository and run createrepo (below). When *that* is deployed, update the puppet manifests to the new versions, omitting the requirements unless their version numbers are important for production.
Now, build a custom repository (or possibly two, one for each architecture) on the distinguished master for the purpose. For example, if you're updating openssh, create a new repos/apt/custom/openssh. If the appropriate repo already exists, use it.
To build a custom repository, start by laying out your package pool. The best plan is to divide the packages by dist. For example:
[root@releng-puppet2.srv.releng.scl3.mozilla.com openssl]# find pool/ pool/ pool/trusty pool/trusty/openssl_1.0.1f-1ubuntu2.5_i386.deb pool/trusty/openssl_1.0.1.orig.tar.gz pool/trusty/libssl1.0.0-dbg_1.0.1f-1ubuntu2.5_i386.deb ... pool/precise pool/precise/libssl1.0.0_1.0.1-4ubuntu5.17_i386.deb ...
Then edit an update.sh in the root of the repository with something along the lines of
for arch in i386 amd64; do
for dist in precise trusty; do
mkdir -p dists/${dist}/all/binary-$arch
dpkg-scanpackages --multiversion --arch $arch pool/$dist > dists/${dist}/all/binary-$arch/Packages
bzip2 < dists/${dist}/all/binary-$arch/Packages > dists/${dist}/all/binary-$arch/Packages.bz2
done
done
This script will update the apt indexes. Note that the `--multiversion` is required if the repo is to contain multiple versions of the same package, like yum repositories can.
Don't forget to run puppetmaster-fixperms afterward to make sure permissions are correct.
If you added a new repository, you'll need to refer to it from the puppet configs. Add a clause to modules/packages/manifests/setup.pp, either a regular packages::aptrepo if the repo should be avialable on every host (like openssh) or a virtual one (prefixed with @) if it should only be available on some hosts. Only installing the repo on some hosts limits the carnage if something goes wrong with the repo.
Then, write or update the classes under modules/packages. If your repository is virtual, you'll need to add something like realize(Packages::Aptrepo['xorg-edgers']) to the package class to ensure the repo is in place.
If you've updated a repo, you need to bump the appropriate counter in modules/packages/manifests/setup.pp:
# to flush the package index, increase this value by one (or # anything, really, just change it). - $repoflag = 5 + $repoflag = 6
This will cause all Ubuntu machines to run apt-get update.
Automatically Pulling Dependencies
Note: This is not at all clear, sorry -- there are no real experts on this topic at Mozilla, so learn what you can and update the wiki!
This needs to be done on an Ubuntu machine. You need to use debpartial-mirror, apt-ftparchive (from apt-utils) and simple wrapper to generate repo indexes:
# http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng-updates.sh debpartial-mirror -c ./releng-updates.conf all rm -rf releng-updates/dists cd releng-updates for arch in i386 amd64; do mkdir -p dists/precise-updates/all/binary-$arch apt-ftparchive --arch $arch packages . > dists/precise-updates/all/binary-$arch/Packages bzip2 < dists/precise-updates/all/binary-$arch/Packages > dists/precise-updates/all/binary-$arch/Packages.bz2 done
[GLOBAL] ;debug = DEBUG mirror_dir = ./ architectures = i386 amd64 components = main distributions = precise get_suggests = true get_recommends = true get_provides = true get_sources = true get_packages = true [releng-updates] server = http://archive.ubuntu.com/ubuntu distributions = precise-updates components = main restricted universe # specify needed packages here filter = name:gnome-settings-daemon
Darwin
Darwin: Repositories
There's no such thing as a repository for OS X packages, sadly. DMGs are stored in repos/DMGs. DMGs are generally built for a specific OS version and put in per-os-version subdirectories, although DMGs that are compatible across versions are in the root. Each DMG is named $packagename-$version.dmg.
Darwin: Installing Packages with Puppet
Use the packages::pkgdmg defined type to install DMGs, giving the package name as the resource name and the package version in the version parameter. The type will construct the correct filename from this information. For example:
Darwin: {
Anchor['packages::wget::begin'] ->
packages::pkgdmg {
wget:
version => "1.12-1";
} -> Anchor['packages::wget::end']
}
DMGs that are downloaded as-is should be documented briefly (at least with the source) in the manifest files that install them. For example (from modules/packages/manifests/puppet.pp):
Darwin: {
# These DMGs come directly from http://downloads.puppetlabs.com/mac/
Anchor['packages::puppet::begin'] ->
packages::pkgdmg {
puppet:
version => $puppet_version;
facter:
version => $facter_version;
} -> Anchor['packages::puppet::end']
}
For DMGs that are *not* os-version-specific, pass os_version_specific => false.
Darwin: Custom Packages
DMGs that are custom built should have a shell script in modules/packages/manifests named $package-dmg.sh which builds the DMG. If there is a corresponding RPM (custom or stock) for the package, then the shell script can require that the source RPM be unpacked first. See ReleaseEngineering/PuppetAgain/HowTo/Build_DMGs for more details.
