Testplan (Overview)

SUMMARY

The purpose of this wiki is to serve as a general testplan for verifying Shavar server deployments

TEST RESULTS

Test results will be posted in Bugzilla within the Shavar server deployment tickets for STAGING and PRODUCTION
- http://mzl.la/1m7oHJH

SIGNOFF CRITERIA

Cloud Services QA team will verify that:

STAGE and PRODUCTION stacks are correctly deployed
List creation scripts are correct
Shavar server is working properly (delivering content to the client)
Sample / Test pages are blocking / unblocking desired content

FUTURE

Identify and test failure cases

Server outage
Partial responses
Erroneous entry on black/whitelist, etc...
Accidental blockage of Firefox updates?

REFERENCE

Shavar Lists

Services/Shavar/Lists

UI for Tracking protection

[UI for tracking protection]

Test Pages

http://itisatrap.org/firefox/its-a-tracker.html
- client-only test! (shavar-server could be completely broken and we wouldn't know)
http://people.mozilla.org/~fmarier/tracking-test/
- page checks if list delivery/generation is broken
- Uses: http://extremetracking.com/ (essentially within an iframe)
- TP OFF: displays black cat bearing its claws: Cat
- TP ON: A fox with a stop sign: Fox

#top

Environment

SHAVAR URLs

STAGE:

shavar.stage.mozaws.net - e2e test / loadtest URL
tracking.stage.mozaws.net - diary study URL
tracking-protection.stage.mozaws.net - CDN for lists

PRODUCTION:

shavar.services.mozilla.com - e2e test URL (no loadtesting on PRODUCTION)
tracking.services.mozilla.com (TO BE DEPRECATED)
tracking-protection.cdn.mozilla.net - CDN for lists

SHAVAR LISTS

There are a number of lists served by the Shavar server that require testing. see: Services/Shavar/Lists

#top

Test Types

Server tests can be roughly broken into the following categories:

Stack Check

Verify that AWS stack has correct instance size, has correct version deployed, is routed correctly, etc.

Loadtesting

Once a stack has been deployed and verified, shavar server can be loadtested to ensure that a release is scalable.
Instance sizing should be performed to identify ideal size to withstand maximum load while maintaining cost-effectiveness of deployment
Shavar server should only be loadtested in staging environment

List Delivery and Creation Test

Follow Setup instructions (create new profile, set prefs!)
Do cache file sanity checks with unchanged black/white lists.

Additions/Deletions Test

Follow Setup instructions (create new profile, set different prefs)
Do cache file sanity checks
Verify that blacklist and whitelist updates get properly delivered to client

#top

Test Tools

There is a bash script (hack) to assist with client/server side testing: https://github.com/rpappalax/shavar-verify

#top

Server-side Test Cases

Stack-Check

PURPOSE

test that the server is configured properly.

STAGE

1) Verify the download of all lists:


 $ curl -d" "  'https://shavar.stage.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:


base-track-digest256
baseeff-track-digest256
basew3c-track-digest256
content-track-digest256
contenteff-track-digest256
contentw3c-track-digest256
mozfull-track-digest256
mozfullstaging-track-digest256
mozplugin-block-digest256
mozplugin2-block-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozstdstaging-track-digest256
mozstdstaging-trackwhite-digest256
moztestpub-track-digest256
moztestpub-trackwhite-digest256

2) Curl each of the list groups.

Example for MOZSTD:


 $ curl -d"mozstd-track-digest256;" 'https://shavar.stage.mozaws.net/downloads?client=foo&appver=1&pver=2.2'

Should return a response similar to this:


 n:3600
 i:mozstd-track-digest256
 u:tracking-protection.stage.mozaws.net/mozstd-track-digest256/1442275596

#top

PRE-PRODUCTION


$ curl -k -d" "  'https://shavarprodtest.prod.mozaws.net/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:


base-track-digest256
baseeff-track-digest256
basew3c-track-digest256
content-track-digest256
contenteff-track-digest256
contentw3c-track-digest256
mozfull-track-digest256
mozfullstaging-track-digest256
mozplugin-block-digest256
mozplugin2-block-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozstdstaging-track-digest256
mozstdstaging-trackwhite-digest256

#top

PRODUCTION


 # shavar URL
 $ curl -d" "  'https://shavar.services.mozilla.com/list?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d" "  'https://tracking.services.mozilla.com/list?client=foo&appver=1&pver=2.2'

Should return all of these lists:


base-track-digest256
baseeff-track-digest256
basew3c-track-digest256
content-track-digest256
contenteff-track-digest256
contentw3c-track-digest256
mozfull-track-digest256
mozfullstaging-track-digest256
mozplugin-block-digest256
mozplugin2-block-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozstdstaging-track-digest256
mozstdstaging-trackwhite-digest256

Curl for the mozstd list:


 # shavar URL 
 $ curl -d"mozstd-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d"mozstd-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'

Should return this:


 n:3600
 i:mozstd-track-digest256
 u:tracking-protection.services.mozilla.com/mozstd-track-digest256/1442275596

Curl for the mozfull lists:


 # shavar URL
 $ curl -d"mozfull-track-digest256;" 'https://shavar.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'
 # tracking URL (CNAME'd to shavar URL - should produce same output)
 $ curl -d"mozfull-track-digest256;" 'https://tracking.services.mozilla.com/downloads?client=foo&appver=1&pver=2.2'

Should return this:


 n:3600
 i:mozfull-track-digest256
 u:tracking-protection.services.mozilla.com/mozfull-track-digest256/1442275596

#top

Loadtesting

SUMMARY

Going forward we will standardize our loadtesting of Shavar server using a fork of the beeswithmachineguns repository which is optimized for our staging environment.

NOTE:

We will also being investigating using Docker containers to test Shavar server on stage. https://hub.docker.com/r/ckolos/bees/tags/

#top

INSTALLATION

Clone the test tool repo: https://github.com/ckolos/beeswithmachineguns/beeswithmachineguns
Follow instructions on README for installation

#top

TEST STRATEGY

In general, we break loadtesting strategy into 2 blocks:

Standalone instance test
1. Find max RPS (requests / second) that 1 EC2 instance can support at desired CPU capacity (and without throwing errors)
2. Result might be recorded as: "standalone instance capacity: X RPS @ Y% CPU"
ELB test
1. Setup auto-scaling parameters in AWS
2. Warm up ELB (see notes below)
3. Kickoff loadtest and allow it to run until scaled to desired number of instances
4. Result might be recorded as: "ELB capacity: X RPS across Y number of instances" (CPU percentage is implicit in auto-scaling parameters)
5. Also make a note of number of concurrent connections needed, 'bee'-size used etc.
6. For our purposes, we will use the largest (HVM instance) bee-size possible to ensure that network is optimized from the bee end (and won't risk bottle-necking there)

NOTES:

EC2 instances must be "warmed-up" prior to a test run
This can be accomplished by throwing a nominal load against test stack for at least 5 minutes just prior to initiating a loadtest run.

#top

EXAMPLE TEST RUN

The following test will:

instantiate -s [# of bees]
of -t [AWS instance size]
using -k <SSH keypair (pem file) name>


 $ cd beeswithmachineguns
 $ ./bees up -s 20 -k loads -t t2.medium

once bees are running, the following test will create:

-c [# of concurrent users]
for a test of duration -w [seconds]
using siege engine --use-siege


./bees attack --use-siege -c 20 -w 300s -u 'http://<shavar-stage-url-here>/downloads POST mozpub-track-digest256;a:1'

or with multiple files:


./bees attack --use-siege -c 20 -w 300s -u 'http://<shavar-stage-url-here>/downloads POST < filelist.txt'

or curl statement to call multiple files:


#!/bin/sh
curl --data-binary @- https://<shavar server>/downloads?pver=2.2 <<EOF
mozstd-track-digest256;a:1
mozstd-trackwhite-digest256;a:1
mozplugin-block-digest256;a:1
EOF

Increase -c and -r from 10 -> 100 -> 500 -> 1000
Monitor the server resources (cpu, memory, network, disk)
Verify the server can handle the load you are applying
Once you reach 60% on any one of those resources, halt test
- note: that is considered 'safe' to account for any spikes/fluctuations that may increase things to 100% usage
- Whatever number of requests per second (RPS) being sent at 60% threshold = amount of load the stack can safely handle

Example test run:

https://bugzilla.mozilla.org/show_bug.cgi?id=1204668#c6

#top

TESTING WITH SEIGE

In Cloud Services, we we'll only use beeswithmachineguns for consistency and optimization around our test environemnt. However, loadtesting of Shavar server can also be done using: seige. See the example below.

The following test will:

create -c [# of concurrent users]
execute -r [# of repetitions (hits)]


siege -b -i -v -c 1000 -r 1000 'https://<shavar-stage-url-here>/downloads?client=LOADTEST&appver=36.0a1&pver=2.2 POST < filelist.txt'

#top

Client-side Test Cases

Setup

SUMMARY

Setup for client-side (end-2-end) verification of shavar server largely consists of:
- creating a new Firefox profile
- setting custom prefs (one set of custom prefs per list-file-under-test)
- verifying some server data is delivered / activated in the client
Below are the custom prefs needed for each of the data lists
Not all data lists (and corresponding prefs) are needed for each test

SET CUSTOM PREFS

NOTE

Prefs are listed below for stage testing.
To test production, replace staging URL (stage.mozaws.net) with production URL (shavar.services.mozilla.com)
all list files are tested in both stage and production (except moztestpub - stage only)
line 9 of "Prefs for: mozpub list file" removes the whitelist ("-trackwhite-") that's in there by default. Once removed it will not be downloaded anymore.

Prefs for: moztestpub list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");           
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");

Prefs for: mozstd list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstd-track-digest256");           
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Prefs for: mozfull list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozfull-track-digest256");            
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Prefs for: mozstaging list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,mozstdstaging-track-digest256,mozfullstaging-track-digest256");                   
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstdstaging-trackwhite-digest256");

Prefs for: base list file

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("browser.safebrowsing.provider.mozilla.lists", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,goog-downloadwhite-digest256,mozstd-track-digest256,mozstd-trackwhite-digest256,mozfull-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256,mozstdstaging-track-digest256,mozfullstaging-track-digest256,mozstdstaging-trackwhite-digest256,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("urlclassifier.trackingTable", "test-track-simple,base-track-digest256,baseeff-track-digest256,basew3c-track-digest256,content-track-digest256,contenteff-track-digest256,contentw3c-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,mozstd-trackwhite-digest256");

Prefs for: moztestpub list file (OLD)

user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("privacy.trackingprotection.enabled", true);
user_pref("urlclassifier.trackingTable", "test-track-simple,moztestpub-track-digest256");
user_pref("urlclassifier.trackingWhitelistTable", "test-trackwhite-simple,moztestpub-trackwhite-digest256");
user_pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,goog-downloadwhite-digest256,moztestpub-track-digest256,moztestpub-trackwhite-digest256");
user_pref("browser.safebrowsing.provider.mozilla.lists", "moztestpub-track-digest256,moztestpub-trackwhite-digest256");

!!!!! restart Firefox !!!!!!

Replace production URL with stage URL for the following prefs:

browser.trackingprotection.updateURL
browser.trackingprotection.gethashURL

Set pref for debugging:

browser.safebrowsing.debug = true
- this will give you some will give you some updates on the console

Remove Google malware & phishing protection:

browser.safebrowsing.phishing.enabled = false
browser.safebrowsing.malware.enabled = false
privacy.trackingprotection.enabled = true (no reboot necessary)

RESET BROWSER

!!!!! restart Firefox !!!!!!

#top

List Delivery & Creation Tests: moztestpub, mozstd, mozfull, mozstaging, base

NOTE:

run tests on both STAGE & PROD (w/ the exception of moztestpub which only exists on STAGE)

list delivery

Verify that all data lists have been delivered and have a reasonable filesize. Execute the following tests, once for each data list.

LIST GROUPS

moztestpub [STAGE ONLY]
mozstd
- mozstd
- mozstdstaging
mozfull
mozstaging
mozplugin
base
- base
- content

verify list files cache

SETUP

Follow setup instructions for all three list files here: Setup

SUMMARY

Once the list updates are done, look into the cache directory and verify: moztestpub, mozstd, mozfull list files should be present and with correct filesize(s).
List updates may not happen immediately after browser restart (see note below).

NOTE:

"The first request for data MUST happen at a random interval between 0 and 5 minutes after the browser starts."
see: https://bugzilla.mozilla.org/show_bug.cgi?id=1045163#c12

FILESIZES

whitelists should be < 400K

mozstd-trackwhite-digest256
mozstdstaging-trackwhite-digest256 [NEW]
moztestpub-trackwhite-digest256

blacklists should be < 100K

mozfull-track-digest256
mozfullstaging-track-digest256
mozstd-track-digest256
mozstdstaging-track-digest256
moztestpub-track-digest256
base-track-digest256

content list < 30k

content-track-digest256

DNT lists < 1k

basew3c-track-digest256
baseeff-track-digest256
contentw3c-track-digest256
contenteff-track-digest256

plugin lists should be > 0

mozplugin-block-digest256
mozplugin2-block-digest256

'CACHE LOCATION The cache has a different location for different operating systems.

Example for moztestpub lists:


    ~/.cache/mozilla/firefox/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Linux 
    ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/moztestpub-track{,white}-digest256.* on Mac

LIST FILES / SIZE All files in the safebrowsing folder created for the profile should have full permissions.

Example for moztestpub lists:


-rw-r--r--   1 johnnyquest  staff   52620 Sep 15 10:55 moztestpub-track-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-track-digest256.pset
-rw-r--r--   1 johnnyquest  staff   59380 Sep 15 10:55 moztestpub-track-digest256.sbstore
-rw-r--r--   1 johnnyquest  staff  321676 Sep 15 10:55 moztestpub-trackwhite-digest256.cache
-rw-r--r--   1 johnnyquest  staff      16 Sep 15 10:55 moztestpub-trackwhite-digest256.pset
-rw-r--r--   1 johnnyquest  staff  362068 Sep 15 10:55 moztestpub-trackwhite-digest256.sbstore

verify its-a-tracker page

repeat test 1 x for each of these list groups:

STAGE: mozstd, mozfull, mozstaging, base
PROD: mozstd, mozfull, mozstaging, base

Note: skip this test for moztestpub-track-digest256.

STEPS:

Pull up this page: http://itisatrap.org/firefox/its-a-tracker.html
Ensure that it is all green
- blacklist and whitelist are working
This page doesn't hit shavar server at all
If not working, then either about:config issue or client issue - repeat above!

verify CNN page

repeat test 1 x for each of:

STAGE: moztestpub, mozstd, mozfull, mozstaging, base
PROD: mozstd, mozfull, mozstaging, base

STEPS:

Pull up CNN: http://www.cnn.com
make sure that the Shield shows up
the developer tool console tab should indicate that something was blocked by tracking protection - should say something like URL blocked because of tracking

verify YouTube page

repeat test 1 x for each of:

STAGE: mozstd, mozfull, mozstaging, base
PROD: mozstd, mozfull, mozstaging, base

Note: skip this test for moztestpub-track-digest256.

STEPS:

Pull up Youtube: https://www.youtube.com
Shouldn't see the shield
Make sure that doubleclick.net is NOT blocked by tracking protection on that site
If the whitelist doesn't work, the Tools > WebDeveloper > BrowserConsole should indicate that doubleclick.net was blocked by TP
If you see in console (network) tab that doubleclick.net is blocked, that's a sign that TP is NOT Working

#top

List Additions / Deletions Tests: moztestpub

NOTE:

STAGE only

blacklist deletions

verify enabled

Follow setup instructions for moztestpub file here: Setup
Pull up this test page: https://people.mozilla.org/~fmarier/tracking-test/
ensure that the resource is blocked:
You should see:
- A fox with a stop sign: Fox
- A tracking protection shield icon in the URL bar
- A message in the: Tools > Web Developer > Browser Console: "The resource at "https://extremetracking.com/" was blocked because tracking protection is enabled."

delete from blacklist

Follow setup instructions for moztestpub file here: Setup
Remove the extremetracking.com json blob from the blacklist (track.json) on https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

verify disabled

Restart Firefox from terminal to force list updates on client
Start Debugging

Verify debugging log (in terminal where you launched Firefox).

Here are some examples of what you might see:

update URL has registered correctly

i.e: listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-track-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Registering whitelist- mozstd-trackwhite-digest256

listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-trackwhite-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Checking for updates on shavar server

listmanager: 09:44:07 GMT-0700 (PDT): checkForUpdates with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Request to shavar server begins

listmanager: 09:44:07 GMT-0700 (PDT): makeUpdateRequestForEntry_: request moztestpub-track-digest256;a:1440776223

Update success for both lists

listmanager: 09:44:07 GMT-0700 (PDT): update success for mozpub-track-digest256,moztestpub-trackwhite-digest256 from https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2: 2700

VERIFY

check that it's no longer blocked by visiting https://people.mozilla.org/~fmarier/tracking-test/
If list with disabled trackingprotection has loaded, you should see: black cat bearing its claws: Cat

#top

blacklist additions

add to blacklist

Add the extremetracking.com json blob back to the blacklist on https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

verify enabled

check that it's blocked again by visiting https://people.mozilla.org/~fmarier/tracking-test/
If list with trackingprotection enabled has loaded, you should see: fox with a stop sign: Fox

#top

whitelist deletions

SUMMARY

This test is similar to blacklist deletions, but now we verify the whitelist update (vs. blacklist update)
Normally googlesyndication.com and doubleclick.net are whitelisted so that they don't get blocked from YouTube
By removing them from whitelist, we should be able to verify that they now get blocked!

verify enabled

Start Debuggging

Restart Firefox from terminal to force list updates on client
The pref set in list delivery (browser.safebrowsing.debug) will begin outputting debug messages 10 - 30 secs after you start browser
see: https://pastebin.mozilla.org/8844272

delete from whitelist

Remove doubleclick.net and googlesyndication.com from the "Google" organization in the resources section of the whitelist (trackwhite.json) - https://github.com/mozilla-services/shavar-test-lists
This will trigger the stage list creation script (Cloud Services QA or Ops team can monitor this in Jenkins)

verify disabled

Verify debugging log (from stdout where you launched Firefox from terminal):

update URL has registered correctly

i.e: listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-track-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Registering whitelist- moztestpub-trackwhite-digest256

listmanager: 09:44:04 GMT-0700 (PDT): registering moztestpub-trackwhite-digest256 with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Checking for updates on shavar server

listmanager: 09:44:07 GMT-0700 (PDT): checkForUpdates with https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2

Request to shavar server begins

listmanager: 09:44:07 GMT-0700 (PDT): makeUpdateRequestForEntry_: request mozpub-track-digest256;a:1440776223

Update success for both lists

listmanager: 09:44:07 GMT-0700 (PDT): update success for moztestpub-track-digest256,moztestpub-trackwhite-digest256 from https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=43.0a1&pver=2.2: 2700

NOTE

if update success fails, rest of test will likely not work!

VERIFY

visit youtube.com
you should see the shield in the url bar (whitelist isn't blocking because deleted)
In the: Tools > Web Developer > Browser Console, you should see that doubleclick.net was blocked because of TrackingProtection

#top

QA/Execution/Web Testing/shavar tp

Contents

Testplan (Overview)

Environment

Test Types

Test Tools

Server-side Test Cases

Stack-Check

STAGE

PRE-PRODUCTION

PRODUCTION

Loadtesting

Client-side Test Cases

Setup

SUMMARY

SET CUSTOM PREFS

List Delivery & Creation Tests: moztestpub, mozstd, mozfull, mozstaging, base

list delivery

verify list files cache

verify its-a-tracker page

verify CNN page

verify YouTube page

List Additions / Deletions Tests: moztestpub

blacklist deletions

verify enabled

delete from blacklist

verify disabled

blacklist additions

add to blacklist

verify enabled

whitelist deletions

verify enabled

delete from whitelist

verify disabled

Navigation menu

Search