QA/Throttling Tests
Contents
Summary
Depending on the update strategy, upon release it is reasonable to expect we need to test update throttling:
- If updates are at 100% you should get a background update
- 0% you should NOT get a background update
- Intermediary value should get background updates only for a percentage of cases that's close to the one set (e.g. it's reasonable for a setting of 25%, to get updates in 15-35% of the cases)
- Throttling testing should be done only on beta and release channels.
Throttling test script
#!/bin/sh
# The following URLS are the 10.0.2 idle-time check urls from
# https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Verifying_Throttling
#
# URLS:
# https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/WINNT_x86-msvc/en-US/release/update.xml
# https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/update.xml
# https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86-gcc3/en-US/release/update.xml
# https://aus3.mozilla.org/update/1/Firefox/10.0.2/20120215223356/Linux_x86_64-gcc3/en-US/release/update.xml
#
# To test use the following syntax in a terminal prompt, where %URL% is one of the above URLs:
# ./throttle.sh URL
#
# This should be something large if we're testing to see if we're partially throttled.
# It doesn't have to be huge if we're testing fully throttled or not throttled at all.
URL=$1
NUM_CHECKS=1000
THROTTLE=0
count=0
attempt=0
percent=0
while [ $attempt -lt $NUM_CHECKS ] ; do
curl -s $URL | grep -q hashValue
if [ $? -eq 0 ] ; then
count=`expr $count + 1`
fi
percent=`expr $attempt \* 100 / $NUM_CHECKS`
attempt=`expr $attempt + 1`
clear
echo "Checking $URL"
echo "Please wait, this will take several minutes...\n"
echo "Successful background update pings: $count / $attempt ($percent% complete)"
done
THROTTLE=`expr $count \* 100 / $attempt`
echo "$URL: $THROTTLE% throttled.\n"
NOTE: The script is meant to run in Linux - for running in Windows, check the instructions below
Running in Linux
The urls to test throttling should look something like this: https://aus5.mozilla.org/update/6/Firefox/55.0.2/20170814072924/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/ISET:NA,MEM:8192/default/default/update.xml Where 55.0.2 is an older version of Firefox than the one you want to update to (in this case, we are testing updates to 55.0.3), and the number after it in the URL is the BuildID of that older version (in this case 55.0.2). The BuildID can be found in the JSON file for the older version, or for 55.0.2, the one specified in: https://archive.mozilla.org/pub/firefox/candidates/55.0.2-candidates/build1/win32/en-US/firefox-55.0.2.json. An alternative is to also install the older version, and then get the BuildID from the about:support page.
NOTE: Make sure you get the correct BuildID from the latest released build - a release can have more than one build
To run the script, simply open a Terminal and run: ./throttle.sh https://aus5.mozilla.org/update/6/Firefox/55.0.2/20170814072924/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/ISET:NA,MEM:8192/default/default/update.xml. Make sure throttle.sh is set as an executable file.
Currently the script is run for 5 different Operating Systems. You can run all 5 scripts in parallel, if you execute them in different terminals. See the Current throttling scripts section from below
Running in Windows
To run the throttling script under Windows, you need to do the following:
- Install cygwin with curl:
- Download the latest cygwin installer
- Start the installer
- Click/enter preferences until you reach the "Select Packages" window.
- Click (+) for Net
- Click the entry for curl. (Make sure you select the checkbox for the Binary)
- Complete the installation
- Verify curl works:
- Go to the directory in which you installed Cygwin (default: C:\cygwin64)
- Double-click "Cygwin.bat" OR search for "cygwin" in the Windows Start menu and click on "Cygwin64 Terminal"
- Run the command "curl" - You should see the following output: curl: try 'curl --help' or 'curl --manual' for more information
- Run the script (can run all 5 in separate cygwin terminals -> ALT+F2 to open a new terminal window):
- Save locally the following Throttle.zip archive, then extract the "throttle.sh" file to your home folder (e.g. C:\cygwin64\home\<username>), or any other destination if desired
- Open a Cygwin Terminal window
- Run the script:
- If the throttle.sh file is saved in C:\cygwin64\home\<username>, then just run: ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/45.0.2/20160407164938/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
- If the throttle.sh file is NOT saved in C:\cygwin64\home\<username>, then cd to the location of the script, and then run: ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/45.0.2/20160407164938/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
Current throttling scripts
Currently (Firefox 55.0.3) the following 5 scripts are used:
Windows 32bit:
Windows 64bit:
Mac:
Linux 32bit:
Linux 64bit:
(don't forget the quotation marks " " for Linux command)
To get the most up-to-date scripts, go to the Test Plan wiki page for the most recent version released (e.g. https://wiki.mozilla.org/Releases/Firefox_55/Test_Plan) and look on the bottom of the page for "Throttle Testing". You should be able to re-use the same commands, just by modifying the test URLs as specified under the Running in Linux section.
Possible results
- 100% Throttling
- Running the scripts should always give the result of 100%
- 0% Throttling
- Running the scripts should always give the result of 0%
- 25% Throttling
- Running the scripts should ideally be very close to 25% (e.g. within 20-30%)
- Running the scripts may yield results varying from 15% to 35% - in this case, can do 2-3 runs to see if results are always somewhere in between these values
- When you get any value that's between +/- 5%, you can consider the test as passed
