CrashKill/2009-11-02

From MozillaWiki
Jump to: navigation, search

Misc

Not really enough users/crash data from the 3.6b1 release last week to tell where we are with crash fixing for 3.6. Still under 2000 crashes per day and only 601 signatures to look at. That's around the same that 3.5.x releases get during during their pre-release period.

  • day total-crashes number-of-signatures for 3.6b1
  • 20091027-crashdata.csv 410 230
  • 20091028-crashdata.csv 453 249
  • 20091029-crashdata.csv 527 266
  • 20091030-crashdata.csv 544 297
  • 20091031-crashdata.csv 1049 418
  • 20091101-crashdata.csv 1468 500
  • 20091102-crashdata.csv 1939 601
  • day total-crashes number-of-signatures for 3.5.4
  • 20091025-crashdata.csv 1852 908
  • 20091026-crashdata.csv 1730 895
  • 20091027-crashdata.csv 2825 1310
  • 20091028-crashdata.csv 34007 8190
  • 20091029-crashdata.csv 79135 15053
  • 20091030-crashdata.csv 92431 16607
  • 20091031-crashdata.csv 91250 16441
  • 20091101-crashdata.csv 105485 18390
  • 20091102-crashdata.csv 108838 18666

Breakpad & Socorro

    • metrics goals and priorities and high level goals
    • storage situation and retention policies
      • raw crash dumps/jsons (used as deferred storage)
        • 7 Days (no backups) - currently stored on a freebsd NAS server - disks are redundant, hardware not redundant.
      • storage of processed crash dumps and jsons (processed)
        • The jsons will be stored for 3 years (haven't deleted any yet) - Currently stored on a sun thumper, disks are redundant, hardware not redundant. Not being backed up currently, but we have a bug in place to back these up (should be done something in the next couple of weeks).
        • The processed raw dumps are on the same freebsd NAS as above. My current plan is to wipe them out after a week.
      • disk space for the database server
        • Currently on equallogic storage arrays.
        • The head is not redundant, but the disk is redundant.
        • This is backed up onto another eql storage volume every night, and WAL is played back on the backup server to keep things current. (not dumped to tape).
      • other storage?
    • Future Storage - two minutes talking about the hadoop/hdfs plan
    • socorro roadmap

Bugs

Rank Stacks Bug Owner Status
1 nsCycleCollectingAutoRefCnt::decr(nsISupports*)
nsEventListenerManager::Release()
nsGlobalWindow::cycleCollection::UnmarkPurple(nsISupports*)
nsPresContext::Release()
nsGlobalChromeWindow::Release()
bug 521750 dbaron Two approaches: (1) need to tell add-on developers to change their ways (bug 521745, bug 521748, bug 521752 (not this signature, but others pointing here), bug 521753) and (2) we can make the most common threadsafety problem crash less (bug 521750)
2 nsGlobalWindow::cycleCollection::UnmarkPurple(nsISupports*) bug 521750 dbaron fixing in 1.9.1.5
3 UserCallWinProcCheckWow bug 501429 jst possibly fixed by never unloading plug-ins (bug 500925)
4 nsEventListenerManager::Release() bug 521750 dbaron fixing in 1.9.1.5
5 RtlpWaitForCriticalSection bug 511757 jst Flash-related
6 RtlpWaitOnCriticalSection bug 511759 jst not-Flash-related; ADR toolbar?
7 _woutput_l bug 511756 dolske likely TrendMicro toolbar
8 gfxWindowsFontGroup::WhichFontSupportsChar(nsTArray<nsRefPtr<FontEntry> > const&, unsigned int) bug 524462 jdaggett regression in 1.9.1.4, fixing in 1.9.1.5
9 nsStyleSet::FileRules(int (*)(nsIStyleRuleProcessor*, void*), RuleProcessorData*) bug 492675 dbaron possible fix landed for 1.9.1.5
10 _PR_MD_SEND bug 467167 sicking, jimm spin off: malware module detection, bug 523350.
11 nsWindow::GetParentWindow(int) bug 470487 jst, jimm fix to-be-landed for 1.9.1.6
12 GraphWalker::DoWalk(nsDeque&) bug 500105 dbaron, peterv possibly land debugging code on trunk to understand crash better; will land on 1.9.2 for b2
13 NPSWF32.dll@0xbc897
14 nsScriptLoader::StartLoad(nsScriptLoadRequest*, nsAString_internal const&) bug 519886 jst, mrbkap Unable to reproduce, but likely wallpaper fix in the bug, ready to land. Correlated with Skype toolbar and hotmail?
15 RtlpCoalesceFreeBlocks bug 519340 dolske caused by older AVG add-on; can't blocklist because add-on uses same version number
16 Flash Player@0x92160 bug 520058 josh crashes aren't URL-specific, module data would be useful here
17 js_Interpret bug 519363 dmandelin most popular subcrash fixed on trunk; sayrer will land on 1.9.2; we need to backport a stack of 5 patches to 1.9.1.
18 nsDocument::RegisterNamedItems(nsIContent*) bug 525276 smaug fix landed for 1.9.1.5
19 nsCycleCollector::MarkRoots(GCGraphBuilder&) bug 437449 dbaron?
20 NPFFAddOn.dll@0x11867 bug 519343 tomcat need to get in contact with AV vendors (in ss' hands)
21 arena_chunk_init bug 515211 dmandelin candidate fix landed on trunk, will monitor to see if it is fixed.
22 HostentBlob_WriteNameOrAlias bug 508292 dolske Windows DNS resolver library crash on Turkish domains. In contact with Microsoft.
23 objc_msgSend | IdleTimerVector bug 509130 smichaud caused by webkit?
24 arena_dalloc_small | arena_dalloc | free | XPT_DestroyArena bug 519356 ctalbert seems related to compatibility mode; spidering a bunch of URLs didn't cause the crash
25 GoogleDesktopMozilla.dll@0x5512 bug 401513
26 js_TraceObject bug 503772 ??
28 PL_DHashTableOperate | free | nsEventListenerManager::AddEventListenerByType(nsIDOMEventListener*, nsAString_internal const&, int, nsIDOMEventGroup*) bug 516113 ? "new" topcrash after we added PL_DHashTableOperate to skiplist
33 DTToolbarFF.dll@0x4bc19 bug 512040 tomcat trying to repro, but still not crashing
39 nsBaseWidget::Destroy() bug 507928 jst, jimm Mac-version fixed in 1.9.1.4; now Windows-only
39 xul.dll@0x348945 bug 525326 joe gif issue, new in 1.9.1.4, fixed in 1.9.1.5
53 RtlAllocateHeap bug 519340 new in top 25
55 GoogleDesktopNetwork3.dll@0x3dfb bug 519344 tomcat trying to find STR - still not crashing
66 nsXULDocument::ResumeWalk() bug 519767 tomcat installed adblock the way as described in the bug - no crash so far. Might be related to something lese ?
70 nsPluginHostImpl::TrySetUpPluginInstance(char const*, nsIURI*, nsIPluginInstanceOwner*) bug 519752 tomcat url testing run... -no crash so far :( -> next step running the complete crash url lists via plugin builds
72 objc_msgSend | CanonIJPDE@0x1531e bug 519451 tomcat printer driver issue; seems fixed by new driver; need to test if new cocoa printing dialogs help this -> Josh think this will fix it, if not there is probably nothing we can do (Tomcat)
80 radhslib.dll@0x3b6f bug 519348 tomcat will blocklist this dll since the add-on is unsupported since 2006
xx nsHttpsHandler::GetProtocolFlags(unsigned int*) bug 519729 dolske correlated with ComputerBild magazine; johnath contacted; out of top 100 now; crashes in the last week
xx NPSWF32.dll@0x77bd0 bug 516780 jst Farmtown flash; need to know when Adobe will ship a fix; no longer in top 100 (crashes in the last week)
xx wcslen bug 514592 dolske Divx associated crash, in contact with DivX folks
xx memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int) bug 519771 sicking added "nsTArray_base::ShiftData..." to Socorro skip list
xx KiFastSystemCallRet bug 514589 sicking Socorro fix landed; only a few of these in the last week
xx @0x0 bug 519616 jrmuizlar Socorro issue is fixed; still a few of these stacks left