Auto-tools/Meetings/2012-10-15

From MozillaWiki
Jump to: navigation, search

Goals Progress

Below are our top level goals. Beneath each goal, note any progress made towards that goal this week.

  • Ensure B2G has all test automation available to prepare for launch
    • [ON TRACK] Deploy B2G Test Automation into TBPL running on Emulator and Panda platforms.
      • We should have pandas deployed with B2G this week in chassis 3 for testing with mochitest/reftest and device pool.
      • We should have emulator tests for WebAPI device emulator and emulator-mochi running this week on cedar - when do we expect to move those to m-c and Ash?
    • [ON TRACK] Maintain B2G dog food update server as well as builds for development and QA while working to hand them off to releng
    • [ON TRACK] Ensure all webAPIs testable through the emulator have comprehensive automated tests
    • [ON TRACK] Continue to aid QA in creating automated tests for B2G
  • Expand and stabilize Test Automation on new and existing device platforms for NEON and ARMv6 support.
    • [ON TRACK] Have APC.io boards chassis-ready -- rooted, smoketested, and working for ARMv6 testing
      • bc and I got the APC.io rooted, figured out some of the issues surrounding the file system. Still unable to run tests at the moment though.
    • [ON TRACK] Monitor deployment of large scale android and B2G panda board testing, aid with bug investigations
    • [ON TRACK] Implement a new pool-based approach for device management in automation to increase stability on all device testing
      • Fixes and improvements to Autophone in preparation for major refactoring to remove pool elements; started coding mozpool service.
    • [ON TRACK] Deploy Autophone with 40 phones and 70% up time
      • 6 phones, 80% uptime from Oct 8-14 inclusive. 1 phone down all week, a second went down mid-way through Oct. 13.
  • Create and deploy performance automation for desktop, fennec, and B2G product benchmarks
    • [ON TRACK] Create Eideticker tests for B2G on panda boards and run them once a day
    • [ON TRACK] Port Eideticker to win32 and run tests there once a day to measure win32 snappiness for desktop
    • [ON TRACK] Create pageload benchmark with networking shaper tests so that we can adequately track user-perceived performance across all devices and network types
    • [ON TRACK] Monitor Datazilla webservices and ensure our accuracy in detecting performance regressions on a per-push basis
    • [ON TRACK] Deploy Datazilla per-push UI to make it easier for developers to dig into the causes of performance regressions
  • Update our bugzilla instance to 4.2 providing many benefits from UI to helping people keep track of bugs/projects from within Bugzilla.
    • [ON TRACK] Complete the 'Sandstone' CSS update
    • [ON TRACK] Complete the REST API integration
    • [ON TRACK] Rework browserID <--> BMO integration
    • [ON TRACK] Deliver user and product dashboards
    • [ON TRACK] Deploy tracking flags extension database optimization
  • Make the process of sheriffing easier by updating TBPL and helping make it easier to handle the intermittent test problem
    • [ON TRACK] Aid with B2G changes so that B2G oranges tracked in Orange factor, B2g tests show up in TBPL
      • [edmorley] Fixed bug 794715 to make TBPL recognise B2G Panda builds & ics_armv7a gecko builds will now display as "B2G Arm" to be more consistent.
    • [ON TRACK] Make it easier to sheriff and diagnose issues through better notification and automatic suggestions of issues for intermittent failures
      • [edmorley] Worked on several bugs in TBPL's BzAPI usage that contribute to the failure summary timeouts, that prevent automatic starring.
      • [edmorley] WIP patch to stop generating annotated summaries for every failure when there are thousands of them (bug 795460).
      • [edmorley] WIP remoteautomation.py patch for outputting the current test filename for Android crashes, so TBPL can correctly look for bug suggestions (bug 757838).
      • OrangeFactor improvements for fixed/disabled tests now in production.
    • [ON TRACK] Split mochitest browser-chrome from mochitest-other to improve parallelism and overall turnaround time.
      • [edmorley] All dependant bugs now landed and deployed, waiting on one last patch to be reviewed (by releng).
    • [ON TRACK] Make orange factor update more than once a day to help make it easier to see changes over time
    • [ON TRACK] Assist with getting Datazilla's UI integrated with TBPL, phase 1
  • Expand and Modernize the QA Automation framework as well as make it easier and more clear how to write automated tests for QA.
    • [ON TRACK] Finish and deploy mozmill 2.0 changes to frameworks and existing QA tests
    • [ON TRACK] Update selenium grid to use ESX on-demand to improve web QA capacity
    • [ON TRACK] Re-organize and improve the automated testing documentation at MDN
    • [ON TRACK] Develop tests for WebRTC, B2G and other high profile projects

Highlights

This is a place for people to put announcements or to describe activities that they'd like to highlight. It's not the responsibility of the MC to find things to put here.

  • Successful integration testing of Black Mobile Magic (Pandaboard reimaging service) last week!
  • glob recently posted: bugzilla.mozilla.org integration best practices. TBPL & OrangeFactor found that virtually all of their uses of BzAPI could be improved -- please check your projects/local scripts to see if you can help reduce load too! :-)

In-Depth Project Discussion

Pick one project/presenter per week who will give a detail presentation on their project.

The Overview

Progress, notes, problems, and other statuses about current projects. Add any new projects to this template!

AutoPhone/Noah's Ark

  • [mcote] Started coding the device-pool (tentatively called "mozpool") service.
  • [mcote] Patch in review to accomplish the following, which will be necessary in mozpool:
    • New DISCONNECTED state to Autophone to indicate device/network errors. Autophone will still try to contact and recover the phone every 15 minutes in this state. DISABLED now means that either the worker subprocess crashed too much (5 or more times within 30 s) or the user issued the "disable" command. No attempts to use the device will be made until a user manually re-enables the device via the "enable" command.
    • Updated to reflect recent mozdevice error-reporting changes.

Bughunter

Bugzilla

  • push : https://globau.wordpress.com/2012/10/11/happy-bmo-push-day-16/
  • [glob] bmo access log analysis and blog post
  • [glob] played whack-a-mole with lots of invalid bugs (evidently filed by training facilities)
  • [glob] finished initial version of tcl integration
  • [glob+dkl] loads of upstream reviews
  • [dkl] more work on the 'project review form'
  • [dkl] minor flagtype, needinfo fixes; backporting of performance fixes
  • [dkl+glob] normal administrative work

Eideticker

  • [wlach]/[mdas]: First cut of eideticker+b2g getting closer...

Firebug Testing

Jetperf

Marionette

  • [mdas] working bug 797912 to get marionette commands into the Marionette JS framework
  • [mdas][jgriffin] working towards getting gaia tests in CI, by making the frameworks more managable for us.

Mobile Automation

  • [ahal] Finished B2G emulator unittest mozharness script
  • [ahal] Getting reftests back up after last week's decimation

Mobile Evangelism Site-Scraping Tool

Mochitest

MozBase

  • [mcote] Some fixes and improvements to mozdevice's 'dm' (formerly 'sut') command-line client.
  • [wlach] / [ahal] - Many mozdevice fixes
  • [vvuk] - Added support for getting SUT devices via mdns to dmSUT: bug 792048

Mozmill

MozTrap

New Graphserver (Datazilla)

Open Web Apps

Peptest

Powerball

Pulse

Signal From Noise

Speedtests

Systems

Talos

Templeton

  • [mcote] Released v 0.6.1 to pypi, just some old-but-unreleased changes.

Sheriffing & Trees

  • [edmorley] Day-to-day sheriffing of the trees.
  • [edmorley] Had a great discussion with Glob in response to the 'bugzilla.mozilla.org integration best practices' post. Found many places in TBPL/OrangeFactor that were not doing the correct thing, see sections below.
  • [edmorley] Discussed unhiding of Valgrind on TBPL & policy about when things can be visible (bug 800435).
  • [edmorley] WIP remoteautomation.py patch for outputting the current test filename for Android crashes, so TBPL can correctly look for bug suggestions (bug 757838).

Treestatus

Tinderboxpushlog

  • [edmorley] New things in production: https://bugzil.la/790963,791643,794768,794951,795298,795476,796934,799564,799908
  • [edmorley] Patches:
    • Client-side-only testing now works without having to edit Config.js each time (bug 799908) - just clone TBPL's repo, run index.html & you are good to test/hack :-)
    • Making TBPL's parser compatible with mozharness/mozlog (bug 799564).
    • Making TBPL recognise B2G Panda builds & ics_armv7a gecko builds consistent with other platforms (bug 794715).
    • Fixing several clownshoes in TBPL's use of BzAPI (which contributes to b.m.o load as well as TBPL log generation timeouts), including:
      • Fetching every comment each time a bug is starred (~300 times a day; and not even using the result correctly *faceplam*).
      • Not specifying desired fields (causing 150 fields to be returned when we need at most 4 of them).
      • Performing single character summary searches (that unsurprisingly timeout) multiple times a day.
      • Full list: https://bugzil.la/799534,799603,800875,799612,800888,799637
    • Fix reftest analyser when testing locally (bug 800337).
    • WIP patches to clean up platform/OS/type regexes.
    • WIP patch to stop generating annotated summaries for every failure when there are thousands of them (bug 795460).
  • [edmorley] Reviewed bug 726514.

War on Orange

  • [edmorley] OrangeFactor work:
    • Shiny new things in production: https://bugzil.la/620509,795843,797320
    • Patch for making client-side-only testing work without having to adjust paths in woo.config.js each time (bug 799950) - just clone OF's repo, run index.html & you are good to hack on the UI :-)
    • Patch to reduce reduce impact of OF's BzAPI calls (bug 799645).

Web QA

Automation Development

  • [henrik] Makefiles and first basic mochitest landed for WebRTC
  • [henrik] First crashtest about to land even if disabled on m-c (will run those tests only on alder]

Upcoming Events

Standing weekly meeting related to our projects should go here

Round Table

  • Meeting format -- still love the no-emcee meeting?
  • Becoming merge-day-aware
  • B2G Tests - Ash, Cedar, Aurora, etc.
  • Work week next week details
    • Documentation hackfest -- seems like there are two different ideas for documenting stuff (1, 2). We should figure this out.

Take notes here and copy and paste them to this section afterwards.

Notes

Holidays and Trips

  • [edmorley] Away Wednesday 17th for BD.

Next week MC

Misc

Find something you think is cool, interesting, funny, or exciting and put it here

Retrieved from "https://wiki.mozilla.org/index.php?title=Auto-tools/Meetings/2012-10-15&oldid=479898"