Mobile/Fennec Unittests/Remote Testing
Contents
Overview
While battling Windows Mobile and python, [we came to the conclusion] that we needed to shift out of our model where we run everything on the device and allow for some or all test tools and data to be run remotely.
This has proven to be a reliable model and will meet our needs with Android. We also have found this very useful for investigating tests on Maemo as we don't need to ssh to the device and do all the setup.
Requirements / Setup
Testing remote requires a few things:
- a platform that you can install an agent on (currently, Windows Mobile (c++), Android (java), and Maemo/Linux (python). NOTE: a generic C++ linux agent is in development
- If you're on Android and you want to use SUTAgent, you must be rooted. You also need to be prepared to manually allow root access to SUTAgent, as it needs to execute a number of commands (most notably kill) which require this level of access.
- As an alternative to manually giving SUTAgent permission to do these things, make a backup copy of /system/bin/su and run 'cat /system/bin/sh > /system/bin/su' as described in the instructions for remote tegra imaging (search for tegra_gainroot.sh).
- a recent build or downloaded tests package (from mozilla-central, not 1.9.2)
- the ip address of your device
- python on your host machine to run the test harness runners
To set this up, you need to first get the test agent for your platform here. Copy that to your device and set it up to run automatically on boot.
- NOTE: fully implemented agents have a GUI and will give you useful information like ip address, commands running, and audible alerts
Next you need to determine your ip address. This can be done with a fully implemented agent, command line tools, or looking at your router.
Once this is running lets test it from your desktop (host):
- telnet <ip> 20701
On some WiFi android devices, the agent might report an IPv6 address that you might not be able to telnet to. You should be able to get an IPv4 address by going to the android network settings screen and clicking on the WiFi network you're connected to.
Now that you have verified you can connect, the next step is to start writing python code. We have a library (devicemanager.py) which allows you to do common (file and process) operations across the telnet interface.
Assumptions
Since we developed this for Mozilla Fennec and Firefox on remote devices, we have made some assumptions which you should be aware of.
- / for all platforms, even windows
- Our agents will convert / to \ if the device uses \. This can cause confusion in commands with arguments such as a browser with http://url. We special case the :// type arguments, so keep that in mind.
- /tests is available as the root of everything
- To simplify things, we assume that /tests is a valid path. If it isn't, we special case that in the agent (as seen in the python agent). In our automation scripts, we rely on /tests for log file paths, profile directories, application, and data.
- install application to /tests/appname
- The existing automation scripts depend on your application (Fennec/Firefox) to be installed in /tests/<appname> (ex. /tests/fennec.) This decision helped simplify things. This is done by pushing and unzipping the application package file to the /tests directory.
- default telnet port is 20701, data channel is 20700.
- in the past we started with 20720, but as we evolved, we have standardized on 20701. We have a data channel defined on port 20700 for the heartbeat and output of some commands (like exec and unzp.) This is important since our integration with buildbot depends on this for monitoring device state.
Available Agent Commands
run [executable] [args] - start program no wait
exec [executable] [args] - start program wait
fire [executable] [args] - start program no wait
arun [executable] [args] - start program no wait
kill [program name] - kill program no path
ps - list of running processes
info - list of device info
[os] - os version for device
[id] - unique identifier for device
[uptime] - uptime for device
[systime] - current system time on device
[screen] - width, height and bits per pixel for device
[memory] - physical, free, available, storage memory for device
[processes] - list of running processes see 'ps'
alrt [on/off] - start or stop sysalert behavior
disk [arg] - prints disk space info
cp file1 file2 - copy file1 to file2 on device
time file - timestamp for file on device
hash file - generate hash for file on device
cd directory - change cwd on device
cat file - cat file on device
cwd - display cwd on device
mv file1 file2 - move file1 to file2 on device
push filename - push file to device
rm file - delete file on device
rmdr directory - delete empty directory on device
prune directory - delete directory on device even if not empty
mkdr directory - create directory on device
dirw directory - tests whether the directory is writable on the device
stat processid - stat process on device
dead processid - print whether the process is alive or hung on device
mems - dump memory stats on device
ls - print directory on device
tmpd - print temp directory on device
ping [hostname/ipaddr] - ping a network device
rebt - reboot device
unzp archivename [dir] - unzip a zipfile respecting directories optionally into dir
clok - print current device time
quit - reset all network connections on SUTAgent
exit - shutdown the SUTAgent
help - you're reading it
DeviceManager library
sendCMD(cmdline, newline = True, sleep = 0) stripPrompt(data) pushFile(localname, destname) mkDir(name) mkDirs(filename) pushDir(localDir, remoteDir) dirExists(dirname) fileExists(filepath) listFiles(rootdir) removeFile(filename) removeDir(remoteDir) getProcessList() getMemInfo() fireProcess(appname) launchProcess(cmd, outputFile = "process.txt", cwd = '') communicate(process, timeout = 600) poll(process) processExist(appname) killProcess(appname) getTempDir() getFile(remoteFile, localFile = '') getDirectory(remoteDir, localDir) validateFile(remoteFile, localFile) getRemoteHash(filename) getLocalHash(filename) getDeviceRoot() getAppRoot() getTestRoot(type) signal(processID, signalType, signalAction) getReturnCode(processID) unpackFile(filename) reboot(wait = False) validateDir(localDir, remoteDir)
Mozilla Unittests
We have successfully run all our automation tests on a remote device. Here are links to the code and so notes about how these are run for anybody extending or debugging.
First off, we did a lot of work to refactor all the existing python harnesses (talos, mochitest, reftest, xpcshell) into classes and smaller methods. This was necessary in order to subclass these scripts so we can overload certain methods and add or remove specific functionality.
In addition to the refactoring, we had to change a lot of the tests to remove the hardcoded localhost and 127.0.0.1 references. This was a large change, but it gave us the option to run all mochitest (plain not chrome), talos, and reftest tests from a remote webserver. This is the recommended way to run all the tests to avoid setting up a webserver on the device.
As a developer working on testing your code or fixing a remote test, please take note that there could be a small issue with a test running remotely vs a real failure on the product.
Mochitest ([runtestsremote.py]):
- this is a subclass of automation.py and runtests.py.
- Overloads the Process class defined in automation.py to use devicemanager.py
- adds extra command line arguments
- --deviceIP, --devicePort, --remote-logfile, --remote-webserver, --http-port, --ssl-port, --remoteProductName
- --remote-logfile requires knowledge of the remote system and the full path to the log file, not /tests. This is embedded in the url and we are not going to be munging that
- assumes fennec is installed in /tests/fennec
- assumes profile in /tests/mochitest
Reftest([remotereftest.py]):
- subclass of automation.py and reftest.py
- Overloads the Process class defined in automation.py to use devicemanager.py
- adds extra command line arguments
- --device, --devicePort, --remoteProductName, --remoteAppName, --remote-webserver
- assumes fennec is installed in /tests/fennes
- assumes reftest.jar is included in the fennec build, not copied to it as we do on desktop
- assumes profile in /tests/reftest
- currently doesn't work with http(...) reftests
Talos([code]):
- retrofitted PerfConfigurator.py and run_tests.py scripts to use devicemanager.py library for os and process calls
- added a winmo platform type for process, setup, and profile management
- profile needs to allow xpconnect for remote host, not localhost
- tested in production for ts, but still need to verify tsvg, tp4, tpan, tzoom
Xpcshell([remotexpcshelltests.py]):
- subclass of runxpcshelltests.py
- adds extra command line arguments
- --device
- assumes fennec is installed in /tests/fennes
- assumes profile in /tests/xpcshell
- will create a .zip of the xpcshell/* tests and copy that to device under /tests/xpcshell
DeviceManager Examples
Copy File to device:
import devicemanager
dm = devicemanager.DeviceManager('192.168.1.103', 20701)
dm.pushFile('/home/joel/config.txt', '/config.txt')
Remote Testing Examples
There as some examples of running remote tests.
From a Build Environment
Note: If you just want to run tests in the recommended fashion, you're probably better off just using the instructions on the main Android page. These instructions are mostly useful if you're trying to work on / understand the internals of remote testing.
This requires a host build to run xpcshell + httpd.js as a webserver (for all but talos). What I normally do is install a build of Fennec on my Android board, then from a desktop build environment (note these have to be the same version so the extensions installed from the desktop build will run on the device), I do this:
- mochitest
cd $(objdir)/_tests/testing/mochitest python runtestsremote.py --deviceIP=<your.device.ip.addr> --xre-path=../../../dist/bin --app=org.mozilla.fennec --test-path=gfx # add a '--dm_trans=adb' if you are connected to your device locally and don't have a SUT agent installed
- reftest
cd $(objdir)/_tests/reftest python remotereftest.py --deviceIP=<your.device.ip.addr> --xre-path=../../dist/bin --app=org.mozilla.fennec tests/layout/reftests/reftest-sanity/reftest.list # add a '--dm_trans=adb' if you are connected to your device locally and don't have a SUT agent installed
- talos
hg clone http://hg.mozilla.org/build/talos #configure apache to point the web root at the talos directory (http://localhost/getInfo.html should resolve) cd talos python remotePerfConfigurator.py -e org.mozilla.fennec --remoteDevice <your.device.ip.addr> --webServer <your.apache.ip.addr> --resultsServer ' ' --resultsLink ' ' --activeTests tpan --output tpan.yml python run_tests.py -d -n tpan.yml # NOTE: we don't have --dm_trans=adb support in talos yet
To Run Tinderbox Style
The buildbot automation uses the SUTAgent to run the tests remotely. It also runs the tests by downloading all the packages it needs from builders. So, this is a good option if you don't have a build environment set up and you need to quickly test some functionality. The first preparation takes a moment, but once you have a prepared directory, it's just a matter of replacing the tests and the build of fennec on the device.
These instructions are for linux, adjust as necessary for other OS's. This should work for mac and windows too.
Test Setup
- Download a nightly linux desktop build of firefox (needed to have all the SO's to run httpd.js)
- Download a nightly linux desktop test bundle of firefox (the rest of the stuff to run httpd.js) and put the required files into your build
- Download or build the fennec you want to install, and install it to your phone. (Use adb, the agent or whatever).
- Download the Fennec test build and extract it
- Install the agent and associated files onto your device (you can use adb for this)
- NOTE: If you have the device attached to USB, you can use the adb devicemanager and you don't need the sut agent.
- NOTE: If this is a personal device you may not want to install Watcher.apk. This will prevent you from using the agent to install/update/uninstall packages from the device. The watcher ensures the agent starts on boot, is always running, disables all power management and alerts if the phone loses network connectivity. So, it's great for automation, but not something you want on your personal device.
- If you're using the agent, you need to start it. You can do this by rebooting the device, or finding it in your applications list. It's called "sutAgentAndroid".
$> wget ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-7.0a1.en-US.linux-i686.tar.bz2 $> tar -jxf firefox-7.0a1.en-US.linux-i686.tar.bz2
$> wget ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/firefox-7.0a1.en-US.linux-i686.tests.zip $> unzip firefox-7.0a1.en-US.linux-i686.tests.zip "bin/*" -d ffxtests $> cp -R ffxtests/bin/* firefox/.
$> wget ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-r7/fennec-7.0a1.multi.eabi-arm.tests.zip $> unzip fennec-7.0a1.multi.eabi-arm.tests.zip -d tests
$> adb install tests/bin/sutAgentAndroid.apk $> adb install tests/bin/Watcher.apk $> adb install tests/bin/FfxCP.apk $> adb install tests/bin/FenCP.apk
Once you have the agent on the device, and you have a prepared firefox build, you just need to download a new fennec and a new set of fennec tests to do new testing. So to actually run some tests, you'll need the IP address of your device, which is displayed on the agent's screen (which should be on your phone if the agent is running).
- Mochitest:
- Add a --dm_trans=adb to the command line if you're not using the sut agent.
$> cd tests
$> python mochitest/runtestsremote.py \
--deviceIP=<your device IP> \
--devicePort=20701 \
--app=org.mozilla.fennec \
--xre-path=../firefox \
--utility-path=../firefox \
--certificate-path=certs \
--test-path=<path to tests you want to run>
- Reftest:
- This applies to crashtests and jsreftests too. Reftest and crashtest cannot run unless you have a very high resolution on your device. This is due to bug 615386. Also, for jsreftest, you need to add the --enable-privilege flag to the command line due to bug 651012
- Add a --dm_trans=adb to the command line if you're not using the sut agent.
$> cd tests
$> python reftest/remotereftest.py \
--deviceIP=<your device IP> \
--devicePort=20701 \
--app=org.mozilla.fennec \
--xre-path=../firefox \
--utility-path=../firefox reftest/tests/<path to manifest>
