Verifying Pings

Testing methodology

1. Set up your Firefox browser to allow for inspecting browser initiated ssl requests
2. Set the desired pref’s for a test case
3. Restart your browser to ensure an accurate starting point for the test
4. Follow test steps
5. Trigger pings of various types (see triggering pings) to validate functionality

Once you have tested against the live servers, you can validate against a local server to get a deeper look at the pings.

1. Set up a local server to accept pings for inspection
2. Set up your Firefox browser to send pings to your local server
3. Trigger pings of various types to validate functionality

Browser Prefs

Preferences documentation

FHR Prefs

• datareporting.healthreport.service.enabled
• datareporting.healthreport.uploadEnabled

UT Prefs

• toolkit.telemetry.enabled
• toolkit.telemetry.unified - enables UT instead of just the old telemetry
• toolkit.telemetry.unifiedIsOptIn - makes Telemetry opt-in even with UT enabled
• toolkit.telemetry.optoutSample - makes Telemetry opt-out for a 5% sample of users if UT is on and opt-in

Triggering Pings

Trigger Telemetry pings

1. Disable a plugin

• This will change your saved browser configuration and trigger a ping
• Note that this is limited to 1 ping every 5 minutes, which may or may not be frequently enough for testing purposes

1. Use the browser console

• In your browser console import TelemetryController: Components.utils.import("resource://gre/modules/TelemetryController.jsm");
• Trigger a simple ping with: TelemetryController.submitExternalPing("foobar", {});

1. Restart your browser

• When the browser is shutdown and then started again, a new telemetry session should begin. You should see saved-session

Trigger FHR pings

1. Update the datareporting.healthreport.servicenextDataSubmissionTime pref to be approx. 60 seconds from ‘now'

• I used http://currentmillis.com/ to grab the current UTC millis and added 60000 to it
• When the millis elapses a ping will be sent to FHR within a minute or so

Client Testing

Pref Control Test Suite

Notes

This suite tests that the behavior of pings (recorded and uploaded) is controlled correctly by the selected browser preferences. This suite should be run against the live servers to make sure the correct endpoints are hit, and that data is or is not sent as expected based on the prefs that are set.

There are a handful of prefs required to effectively test telemetry pings and behavior. See the validating pings section for more info and set up instructions.

The two endpoints that pings should be sent to are https://fhr.data.mozilla.com/1.0/submit/ and https://incoming.telemetry.mozilla.org/submit/

For Old Telemetry and Unified Telemetry, all data is sent to the incoming.telemetry URL with the parameter ?v=4. OT sends "saved-session" pings, UT sends "main" pings. The ping types can be inferred from the ping request URL, e.g. https://incoming.telemetry.mozilla.org/submit/telemetry/14f9e835-d100-b44d-b833-8c027d220453/main/Firefox/42.0a1/nightly/20150721030212?v=4 is a /main/ ping as identified after the ID. A saved-session would look something like https://incoming.telemetry.mozilla.org/submit/telemetry/78eff6fd-40ca-6742-8630-5dbbff090e89/saved-session/Firefox/42.0a1/nightly/20150721030212?v=4.

Test Cases

1. Everything disabled

Initial State: Clean start up with everything disabled, upload enabled

Steps:

• Trigger all pings

Expected result: No data should be sent to either fhr.data or incoming.telemetry

2. Everything enabled

Initial State: Clean start up with everything enabled, upload enabled

Steps:

• Trigger all pings

Expected result: Data is sent to fhr.data, main and saved-session pings are sent to incoming.telemetry

3. FHR and OT enabled, UT disabled

Initial State: clean start up with FHR and OT enabled, upload enabled, UT disabled

Steps:

• Trigger all pings

Expected result: Data is sent to fhr.data and saved-session pings are sent to incoming.telemetry

4. FHR enabled, but upload disabled

Initial State: clean start up with FHR enabled, upload disabled

Steps:

• Trigger all pings

Expected result: no data sent to either fhr.data or incoming.telemetry

5. Everything enabled, but upload disabled

Initial State: clean start up with everything enabled, upload disabled

Steps:

• Trigger all pings

Expected result: no data sent to fhr.data or incoming.telemetry

6. Everything enabled, confirm UT is sending, then disable UT

Initial State: Active instance with everything enabled, upload enabled

Steps:

• Trigger all pings
• Confirm incoming.telemetry is receiving main pings
• Disable UT

Expected result: incoming.telemetry stops receiving main pings, saved-session pings continue to send

7. Everything enabled, confirm UT is sending, then disable upload

Initial State: Active instance with everything enabled, upload enabled

Steps:

• Trigger all pings
• Confirm incoming.telemetry and fhr.data is receiving main pings
• Disable upload

Expected result: fhr.data no longer receives data

8. FHR and OT enabled, confirm OT is sending, then enable UT

Initial State: Active instance with FHR and OT enabled, UT disabled, upload enabled

Steps:

• Trigger all pings
• Confirm incoming.telemetry is receiving saved-sessoin pings but not main pings
• Enable UT

Expected result: Data is sent to fhr.data and incoming.telemetry receives both saved-session and main pings

9. Disabling FHR upload results in a delete request being sent

Initial State: FHR and Upload prefs are enabled

Steps:

• Trigger all pings
• Confirm fhr.data and incoming.telemetry are receiving pings
• Disable upload

Expected result: Delete request is sent to https://fhr.data.mozilla.com/1.0/submit/

Ping Type Test Suite

Notes

There are several types of pings to look at: https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/pings.html#ping-types

Additionally under the main ping type, there are several reasons: https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/main-ping.html

Each of these ping types has specific conditions that trigger them, as outlined in the docs. These cases have been covered in this test suite.

The two main types of pings are the old saved-session ping, and the new main ping. The old saved-session ping will soon be removed, and will be handled by UT's main ping with a reason of saved-session.

main ping reasons: aborted-session environment-change shutdown daily saved-session

Session and Subsession Validation Test Suite

Server Testing

The data pipeline testing involves validating the accuracy and amount of data processed to ensure the incoming pings are counted correctly so that the graphs we produce and make decisions from are reliable.

Testing the servers involves 3 parts:

1. Pipeline Execution
2. Data validation
3. Load

Pipeline Execution

• Sample Data Generator
• Script/tool to submit sample data for processing, possibly based on test/load profiles
• Verify the pipeline successfully processes the data and no errors are raised

Data validation

• Sample Data Generator
• Script/tool to submit sample data for processing, possibly based on test/load profiles
• Analyze and compare actual output to expected output (based on script/sample data input)

Load Testing

• Load is expected to be about 1000rps in beta spiking to 2000 during peak times, with each request being approx. 100kb
• Load is expected to be about 3000rps in production spiking to 8000 during peak times, with each request being approx. 100kb

Inspecting Browser Initiated SSL Requests

pro-tip: if starting fx from the command line you can add the -jsdebugger switch to auto-open the toolbox

1. Open the inspector (option-command-i on mac)
2. Click the gear/settings icon
3. Scroll all the way down to advanced settings (should be on the bottom right)
4. Enable both remote and chrome debugging
5. In about:config make sure devtools.debugger.prompt-connection is set to false
6. Restart firefox
7. In the customize menu there is a developer menu (the wrench, move it to your toolbar for quick access if you want)
8. In the developer menu open the Browser Toolbox, alternatively you can use Tools -> Web Developer -> Browser Toolbox
9. View and use the network tab as you normally would in the regular inspector, except now you can capture events from browser level requests (pocket, sync, telemetry, etc.), including SSL