CrashKill/2009-11-02
From MozillaWiki
Misc
Not really enough users/crash data from the 3.6b1 release last week to tell where we are with crash fixing for 3.6. Still under 2000 crashes per day and only 601 signatures to look at. That's around the same that 3.5.x releases get during during their pre-release period.
- day total-crashes number-of-signatures for 3.6b1
- 20091027-crashdata.csv 410 230
- 20091028-crashdata.csv 453 249
- 20091029-crashdata.csv 527 266
- 20091030-crashdata.csv 544 297
- 20091031-crashdata.csv 1049 418
- 20091101-crashdata.csv 1468 500
- 20091102-crashdata.csv 1939 601
- day total-crashes number-of-signatures for 3.5.4
- 20091025-crashdata.csv 1852 908
- 20091026-crashdata.csv 1730 895
- 20091027-crashdata.csv 2825 1310
- 20091028-crashdata.csv 34007 8190
- 20091029-crashdata.csv 79135 15053
- 20091030-crashdata.csv 92431 16607
- 20091031-crashdata.csv 91250 16441
- 20091101-crashdata.csv 105485 18390
- 20091102-crashdata.csv 108838 18666
Breakpad & Socorro
- metrics goals and priorities and high level goals
- storage situation and retention policies
- raw crash dumps/jsons (used as deferred storage)
- 7 Days (no backups) - currently stored on a freebsd NAS server - disks are redundant, hardware not redundant.
- storage of processed crash dumps and jsons (processed)
- The jsons will be stored for 3 years (haven't deleted any yet) - Currently stored on a sun thumper, disks are redundant, hardware not redundant. Not being backed up currently, but we have a bug in place to back these up (should be done something in the next couple of weeks).
- The processed raw dumps are on the same freebsd NAS as above. My current plan is to wipe them out after a week.
- disk space for the database server
- Currently on equallogic storage arrays.
- The head is not redundant, but the disk is redundant.
- This is backed up onto another eql storage volume every night, and WAL is played back on the backup server to keep things current. (not dumped to tape).
- other storage?
- raw crash dumps/jsons (used as deferred storage)
- Future Storage - two minutes talking about the hadoop/hdfs plan
- socorro roadmap
Bugs
Rank | Stacks | Bug | Owner | Status |
1 | nsCycleCollectingAutoRefCnt::decr(nsISupports*) nsEventListenerManager::Release() nsGlobalWindow::cycleCollection::UnmarkPurple(nsISupports*) nsPresContext::Release() nsGlobalChromeWindow::Release() |
bug 521750 | dbaron | Two approaches: (1) need to tell add-on developers to change their ways (bug 521745, bug 521748, bug 521752 (not this signature, but others pointing here), bug 521753) and (2) we can make the most common threadsafety problem crash less (bug 521750) |
2 | nsGlobalWindow::cycleCollection::UnmarkPurple(nsISupports*) | bug 521750 | dbaron | fixing in 1.9.1.5 |
3 | UserCallWinProcCheckWow | bug 501429 | jst | possibly fixed by never unloading plug-ins (bug 500925) |
4 | nsEventListenerManager::Release() | bug 521750 | dbaron | fixing in 1.9.1.5 |
5 | RtlpWaitForCriticalSection | bug 511757 | jst | Flash-related |
6 | RtlpWaitOnCriticalSection | bug 511759 | jst | not-Flash-related; ADR toolbar? |
7 | _woutput_l | bug 511756 | dolske | likely TrendMicro toolbar |
8 | gfxWindowsFontGroup::WhichFontSupportsChar(nsTArray<nsRefPtr<FontEntry> > const&, unsigned int) | bug 524462 | jdaggett | regression in 1.9.1.4, fixing in 1.9.1.5 |
9 | nsStyleSet::FileRules(int (*)(nsIStyleRuleProcessor*, void*), RuleProcessorData*) | bug 492675 | dbaron | possible fix landed for 1.9.1.5 |
10 | _PR_MD_SEND | bug 467167 | sicking, jimm | spin off: malware module detection, bug 523350. |
11 | nsWindow::GetParentWindow(int) | bug 470487 | jst, jimm | fix to-be-landed for 1.9.1.6 |
12 | GraphWalker::DoWalk(nsDeque&) | bug 500105 | dbaron, peterv | possibly land debugging code on trunk to understand crash better; will land on 1.9.2 for b2 |
13 | NPSWF32.dll@0xbc897 | |||
14 | nsScriptLoader::StartLoad(nsScriptLoadRequest*, nsAString_internal const&) | bug 519886 | jst, mrbkap | Unable to reproduce, but likely wallpaper fix in the bug, ready to land. Correlated with Skype toolbar and hotmail? |
15 | RtlpCoalesceFreeBlocks | bug 519340 | dolske | caused by older AVG add-on; can't blocklist because add-on uses same version number |
16 | Flash Player@0x92160 | bug 520058 | josh | crashes aren't URL-specific, module data would be useful here |
17 | js_Interpret | bug 519363 | dmandelin | most popular subcrash fixed on trunk; sayrer will land on 1.9.2; we need to backport a stack of 5 patches to 1.9.1. |
18 | nsDocument::RegisterNamedItems(nsIContent*) | bug 525276 | smaug | fix landed for 1.9.1.5 |
19 | nsCycleCollector::MarkRoots(GCGraphBuilder&) | bug 437449 | dbaron? | |
20 | NPFFAddOn.dll@0x11867 | bug 519343 | tomcat | need to get in contact with AV vendors (in ss' hands) |
21 | arena_chunk_init | bug 515211 | dmandelin | candidate fix landed on trunk, will monitor to see if it is fixed. |
22 | HostentBlob_WriteNameOrAlias | bug 508292 | dolske | Windows DNS resolver library crash on Turkish domains. In contact with Microsoft. |
23 | objc_msgSend | IdleTimerVector | bug 509130 | smichaud | caused by webkit? |
24 | arena_dalloc_small | arena_dalloc | free | XPT_DestroyArena | bug 519356 | ctalbert | seems related to compatibility mode; spidering a bunch of URLs didn't cause the crash |
25 | GoogleDesktopMozilla.dll@0x5512 | bug 401513 | ||
26 | js_TraceObject | bug 503772 | ?? | |
28 | PL_DHashTableOperate | free | nsEventListenerManager::AddEventListenerByType(nsIDOMEventListener*, nsAString_internal const&, int, nsIDOMEventGroup*) | bug 516113 | ? | "new" topcrash after we added PL_DHashTableOperate to skiplist |
33 | DTToolbarFF.dll@0x4bc19 | bug 512040 | tomcat | trying to repro, but still not crashing |
39 | nsBaseWidget::Destroy() | bug 507928 | jst, jimm | Mac-version fixed in 1.9.1.4; now Windows-only |
39 | xul.dll@0x348945 | bug 525326 | joe | gif issue, new in 1.9.1.4, fixed in 1.9.1.5 |
53 | RtlAllocateHeap | bug 519340 | new in top 25 | |
55 | GoogleDesktopNetwork3.dll@0x3dfb | bug 519344 | tomcat | trying to find STR - still not crashing |
66 | nsXULDocument::ResumeWalk() | bug 519767 | tomcat | installed adblock the way as described in the bug - no crash so far. Might be related to something lese ? |
70 | nsPluginHostImpl::TrySetUpPluginInstance(char const*, nsIURI*, nsIPluginInstanceOwner*) | bug 519752 | tomcat | url testing run... -no crash so far :( -> next step running the complete crash url lists via plugin builds |
72 | objc_msgSend | CanonIJPDE@0x1531e | bug 519451 | tomcat | printer driver issue; seems fixed by new driver; need to test if new cocoa printing dialogs help this -> Josh think this will fix it, if not there is probably nothing we can do (Tomcat) |
80 | radhslib.dll@0x3b6f | bug 519348 | tomcat | will blocklist this dll since the add-on is unsupported since 2006 |
xx | nsHttpsHandler::GetProtocolFlags(unsigned int*) | bug 519729 | dolske | correlated with ComputerBild magazine; johnath contacted; out of top 100 now; crashes in the last week |
xx | NPSWF32.dll@0x77bd0 | bug 516780 | jst | Farmtown flash; need to know when Adobe will ship a fix; no longer in top 100 (crashes in the last week) |
xx | wcslen | bug 514592 | dolske | Divx associated crash, in contact with DivX folks |
xx | memmove | nsTArray_base::ShiftData(unsigned int, unsigned int, unsigned int, unsigned int) | bug 519771 | sicking | added "nsTArray_base::ShiftData..." to Socorro skip list |
xx | KiFastSystemCallRet | bug 514589 | sicking | Socorro fix landed; only a few of these in the last week |