ReleaseEngineering/OPSI
Contents
- 1 Definitions
- 2 Servers
- 3 Install OPSI on a client
- 4 Using/writing OPSI in a nutshell
- 5 Creating a package
- 6 Troubleshooting
- 6.1 Check which master
- 6.2 Check the hostkey
- 6.3 Fix the hostkey
- 6.4 Understanding the key generation
- 6.5 Check the opsiconfd log
- 6.6 uib.local VS .build.mozilla.org
- 6.7 Re-installing OPSI
- 6.8 Mit Netzlaufwerken verbinden, bitte noch etwas warten
- 6.9 Access violation (zugriffsverletzung bei adresse)
- 6.10 error: HTTP/1.1 401 Unauthorized
- 6.11 Wrong hostkey
- 6.12 How to update the opt_pcbin drive mapping password - "pcpatch"
- 7 Documentation/Links
Definitions
- OPSI Server - Serves opsi packages, instructs clients on which packages to install, remove, etc. This is a Linux machine (debian currently).
- OPSI Client - Runs on win32 machines. Runs right before the login prompt at boot and audits the system to make sure the packages are in the proper state. This is when new installations happen.
- OPSI Package - Contains to-be-installed files as well as installation instructions. Sources for these package are to be kept in version control. Packages themselves are kept on the OPSI Server.
- OPSI Config Editor - This is a Java app that is used to manage what is installed on which machines. It lets you schedule installations of packages, and see what is currently installed. (Other things too, but not all packages support them.) You can download it here.
- If you are on OS X and can't start OPSI.jar, launch 'Java Preferences' and change Java 6 64-bit to be the top of the list of runtimes for applications
Servers
We run two OPSI servers:
- staging-opsi.build.mozilla.org
- production-opsi.build.mozilla.org
Install OPSI on a client
Enable File and Print Sharing
- Enable File & Print Sharing
- Control Panel -> Network Connections -> Local Area Connection -> Properties
- Enable "Client for Microsoft Networks" and "File and Print Sharing", click Close.
- Open firewall (only to opsi server)
- Control Panel -> Windows Firewall -> Exceptions
- Enable "File and Print Sharing", click "Edit"
- For each entry:
- Click "Change Scope"
- Select "Custom List"
- Enter the IP address of the OPSI server. (For staging: 10.26.48.39, for production: 10.26.48.38).
OPSI Prep and Install
We need to flip a few options in Windows to enable certain packages to install without prompting.
- Do not prompt for unsigned drivers (needed for at least the windows mobile sdk)
- Control Panel -> System -> Hardware -> Driver Signing, choose "Ignore".
- Security Settings
- Start Menu -> Run -> \\$ipOfOpsiServer (staging: \\10.26.48.39, production: 10.26.48.38)
- Use 'pcpatch' and the root password when prompted.
- Right click 'opt_pcbin' and select 'Mount as Network Drive' and use Z: (this is not a permanent mount, so the letter really doesn't matter)
- Browse to z:\install\preloginloader
- Double click 'opsi-prep.vbs'
- Double click 'service-setup.cmd', hit a key when prompted
- When prompted for a username and password use 'root' and the opsi server's root pw.
Once OPSI is finished installing it will forcibly reboot the system. In a few minutes it should come back up and the client should be listed in the OPSI GUI.
Autoit
We require Autoit to deploy many packages. To install it, do the following:
- Checkout the installer:
cvs -d:ext:cltbld@cvs.mozilla.org:/mofo co ref-platforms/win32/autoit-v3-setup.exe
- (as Administrator), run the installer, accept the defaults
Using/writing OPSI in a nutshell
Installing a package on a client
- Open up the OPSI Config Editor
- Select a client or clients, click on "Product Configuration"
- Change the action to "setup" for anything you want to install. (NOTE the column for "action required")
- The package(s) will be installed at the next reboot.
Writing packages
There are different things you can do depending of what type wInst target you are writing for:
- Do you want to copy a file?
- Then look at some examples of [DosInAnIcon_foo]
- Do you want to handle contents inside of a directory?
- Then look at some examples of [Files_foo]
- Do you want to do registry modifications?
- Then look at some examples of [Registry_foo]
- Do you want to install an MSI silently?
- Then look at some examples of [Winbatch_install_foo]
- and some msiexec examples
- Do you want to install an installer that requires user interaction?
- Then look at some autoit.exe examples
Creating a package
Parts of the package
An OPSI package is built from a directory with a specific layout and set of files. These are kept in version control and should be reviewed before check in. The important files in the package are as follows:
- OPSI/control - This contains the metadata for a package: name, version number, path to install script, etc.
- CLIENT_DATA/packagename.ins - This is the meat of a package. All of the installation work and setup should be done here. It is written in a language called wInst which is fairly well documented.
Less commonly used files:
- OPSI/preinst - Executes before the installation script (shell)
- OPSI/postinst - Executes after the installation script (shell)
Any installers or files a package depends on should be placed in the CLIENT_DATA/ directory, too. That entire directory is downloaded to the client before the installation script executes. Files in it can be referenced from within the installation script.
Bootstrapping a new package
To get the basic files/directories needed to start a new package you should do the following:
hg clone http://hg.mozilla.org/build/opsi-package-sources cd opsi-package-sources rsync -av newproduct-template/ myproduct/
Next:
- Edit OPSI/control with the product details.
- Put package files in CLIENT_DATA/.
- Create an installation script.
Installation script basics
This document doesn't cover wInst syntax in detail, please refer to the wInst manual for that.
Installing an MSI
This is by far the easiest type of package to install. Almost all MSIs have a silent mode which you can invoke. For example, here is the installation script for Visual Studio 2005:
[Initial] Message=installing Visual Studio 2005 ... StayOnTop=false [Aktionen] Winbatch_product_silent_install [Winbatch_product_silent_install] msiexec.exe /i %SCRIPTPATH%\vs_setup.msi TRANSFORM=%SCRIPTPATH%\Setup\vs.ini /passive /l* c:\visualstudioinstall.log
Installing a package without a silent install
A lot of packages don't offer silent install options. To work around this, we depend on a tool called Autoit. Autoit is a tool which automates mouse clicks and keyboard strokes. Here's how to create a package that uses Autoit:
- Use the Autoit "Script Writer" to record the necessary clicks and strokes to install your package. Try to keep unnecessary mouse movement to a minimum while recording to avoid bloating the script.
- Test the script to make sure it works.
- Put the script and the package in the CLIENT_DATA/ directory.
- Create the OPSI installation script and call Autoit from it. For example, here's the installation script for MozillaBuild 1.3:
Initial] Message=MozillaBuild 1.3 Installation StayOnTop=false [Aktionen] Winbatch_install_mozillabuild [Winbatch_install_mozillabuild] autoit3.exe %SCRIPTPATH%\mozillabuild.au3 %SCRIPTPATH%\MozillaBuildSetup-1.3.exe
Updating registry keys
The wInst language provides a nice built-in way to update registry keys as part of an installation. Rather than (or in addition to) calling out to an installer you can write registry keys in the installer. For example, here's how to update the system PATH to append "d:\mozilla-build\python25\scripts":
[Initial] Message=Installing new registry keys StayOnTop=false [Aktionen] Registry_AddPath [Registry_AddPath] openkey [HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment] supp "Path"; "d:\mozilla-build\python25\scripts"
Testing the package on staging
On staging you want to test out the package first to make sure it installs correctly before it goes into production. To create a .opsi package and inform OPSI about it, do the following: (For staging):
# as root on the opsi server cd ~cltbld/opsi-package-sources su -c './regenerate-package dir-with-your-package' # to confirm that what you want installed on the slave is correct # look in: /opt/pcbin/install/dir-with-your-package
If there are any issues and you need to regenerate the package you can simply drop in the new code and run the above again.
Installing the package file on production
Please note that you should NOT install packages onto the production OPSI server unless they have been properly reviewed and checked in. That means your package should be landed in opsi-package-sources (hg), and you should land any binaries in opsi-binaries as cltbld@staging-opsi (CVS).
(For production):
# as cltbld on the opsi server # update binaries cd ~cltbld/opsi-binaries cvs up -Pd # update the package source clone cd ~cltbld/opsi-package-sources hg id && hg in -v hg pull && hg up ./sync-binaries # Recreate and install the package su -c './regenerate-package package-name'
Your package should now show up in the OPSI Config Editor. Make sure to set staging slaves to install/update the package as well as production ones.
Troubleshooting
Check which master
- Open regedit and browse to HKEY_LOCAL_MACHINE\SOFTWARE\opsi.org\pcptch
- "opsiServiceURL" key should point to XXX for production to "https://10.26.48.38:4447" or "https://10.26.48.39:4447" for staging.
Check the hostkey
- Check that the hostkeys on the master and the slave are the same
- in master: grep talos-r3-xp-066 /etc/opsi/pckeys
- in slave: cat C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg
- The value of the slave should be the same as on the master. See Understanding the key generation section below.
- The slave's hostkey is generated with a service on boot.
- OPSI package -> hostkey.ins for Win2003 slaves
schtasks /create /tn opsikey /tr d:\\mozilla-build\\python25\\python.exe d:\\tools\\buildfarm\\opsi\\regenerate-hostkey.py /sc ONSTART /ru SYSTEM
- If it does not match check Fix the host key section.
Fix the hostkey
I've seen before XP slaves not running the regenerate-hostkey.py service and require you to run it manually:
C:\>cat "C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg" [shareinfo] pckey = 80dc2294bff215b3fab57d8e7ff61ccd c:\>del "C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg" C:\>C:\mozilla-build\python25\python.exe C:\tools\buildfarm\opsi\regenerate-hostkey.py C:\>cat "C:\Program Files\opsi.org\preloginloader\cfg\locked.cfg" [shareinfo] pckey = db4bb6ed05f5359cf5ef23870fea3f93
NOTE: you must delete the key file, or the script will not regenerate it! NOTE: the script logs to c:\tmp\key-generator.log
After rebooting you should be good to go
Understanding the key generation
- Both locked.cfg and /etc/opsi/pckeys have to match and both are generated with the same code from two different scripts
- md5(str).hexdigest() look-for-new-slave.py#l82 and regenerate-hostkey.py#l18
- On the master look-for-new-slaves.py is called and adds an entry like this to /etc/opsi/pckeys like this:
talos-r3-xp-067.uib.local:db4bb6ed05f5359cf5ef23870fea3f93
- On the slave regenerate-hostkey.py is run and locked.cfg is generated:
[shareinfo] pckey = db4bb6ed05f5359cf5ef23870fea3f93
If your slave was to match any of these values then it means that you have the key for one of the ref machines and it has to be fixed:
>>> md5('win2k3-ref-img').hexdigest() '9b4cefa1c3a1f9159cf51a6d28155be8' >>> md5('win32-ix-ref').hexdigest() 'c1cdfb2077205f086680ada04310f8c2' >>> md5('talos-r3-xp-ref').hexdigest() '80dc2294bff215b3fab57d8e7ff61ccd'
Look at this code to understand the uniqueness of the value:
>>> from md5 import md5 >>> import sys >>> from socket import gethostname >>> gethostname() 'talos-r3-xp-067' >>> md5(gethostname()).hexdigest() 'db4bb6ed05f5359cf5ef23870fea3f93' >>> md5('talos-r3-xp-ref').hexdigest() '80dc2294bff215b3fab57d8e7ff61ccd'
Check the opsiconfd log
- select the slave from the OPSI.jar menu
- select the tab that says "Log files"
- select the sub-tab named "opsiconfd"
- go to the bottom to see the latest messages
uib.local VS .build.mozilla.org
bhearsum: it indicates what domain the OPSI server thinks it's in. it's long standing that it's different between production and staging, but it shouldn't be an issue
Re-installing OPSI
This is not needed anymore. Check with bhearsum for details.
NOTE: This instructions were build up from this page.
- change the opsiServiceURL to point to staging (explained above)
- In c:\program files\opsi.org\preloginloader\uninst run deinstall.cmd
- the slave will reboot
- disconnect the mapped drive for production opsi
- install OPSI from staging-opsi
- Start Menu -> Run -> \\10.26.48.39
- Use 'pcpatch' and the OPSI password when prompted.
- Right click 'opt_pcbin' and select 'Mount as Network Drive'
- Browse to z:\install\preloginloader
- Double click 'opsi-prep.vbs' (you won't see anything happen)
- Double click 'service-setup.cmd', hit a key when prompted
- When prompted for a username and password use 'root' and the appropriate password (the current one unlike for "pcpatch")
- Press any key (I assume it should have rebooted after this but it didn't - armenzg)
[really? it didn't happen to me - armenzg] OPSI forces reboot on completion of install and then you can manage it through the opsi client.
Mit Netzlaufwerken verbinden, bitte noch etwas warten
It translates to: Connecting to network drives, please wait a bit
It takes several minutes until it times out.
This is a DNS problem which was spotted on bug 620517.
To fix it you have to add build.mozilla.org to the main DNS server as explained in here.
Access violation (zugriffsverletzung bei adresse)
It is a blocking message in a "wInst-Message" pop up saying: "zugriffsverletzung bei adresse <memory address> in Modul 'winst32.exe'. Lesen von Adresse 0000000"
See bug 680457
error: HTTP/1.1 401 Unauthorized
This error appears before logging in. I believe you can only stop "the installation" and be able to access the machine.
bhearsum: This usually means that the slave is talking to the wrong opsi server, or that the hostkey on the slave doesn't match the one the server.
I assume we have to check the following:
- the slave is talking to right master - Check which master
- if the hostkey is the same on the slave and on the master - Check the hostkey
Wrong hostkey
Failed to process rpc: opsiHostKey authentication failed for host 'talos-r3-xp-001.build.mozilla.org': wrong key (opsiconfd|623)
- Check the hostkey section to fix it.
How to update the opt_pcbin drive mapping password - "pcpatch"
NOTE: this has never been tested.
In bug 519989, we did some preliminary investigation into how we might go about updating the password used for mounting the OPSI shared package drive in case we happen to expose it:
- Arrange for a long downtime. Depending about how comfortable you are about burning builds in progress, you'll likely need between 3-6 hours.
- Shutdown, possibly gracefully, any affected buildbot masters.
- Push out an OPSI product that changes the password but leaves the pcpatch user password as it is (for now)
- Reboot the clients, and during their OPSI check-in the change is made. The change is made after the mount is authenticated, so they *should* be fine this time around.
- After all the clients have the new package (as verified through the OPSI dashboard), change the pcpatch user password on the server. Future reboots will work with the new password.
- Reboot the clients again to make sure they can connect.
Further background can be found in bug 519989.