Necko/MobileCache/Tp4m
Contents
Test Setup
In order to figure out if the disk cache was still a performance loss on mobile, we wanted to run a tp4m that would actually make good use of the cache now that all cache I/O has been moved off the main thread. We did this by changing the manifest of pages for tp4m and running it over the MV office wifi on 3 devices (Galaxy Tab 10.1, Samsung Galaxy S (AT&T version), and HTC Sensation) as well as simulating the network conditions using NeckoNet on a desktop (the desktop in question has 4 hyperthreaded cores, 16G RAM, and 256G SSD).
All tests were run using a 100MB cache, with the addition of using the default max cache size of 1GB on desktop as a baseline performance number.
Each run of the test started with an empty cache.
The manifest used for these tests is groups of 4 pages, each group loaded 3 times before moving on to the next group (the URLs were slightly modified for the desktop version)
http://localhost/page_load_test/mobile_tp4/news.google.com/news.google.com/index.html http://localhost/page_load_test/mobile_tp4/m.news.google.com/news.google.com/index.html http://localhost/page_load_test/mobile_tp4/amazon.com/www.amazon.com/index.html http://localhost/page_load_test/mobile_tp4/m.amazon.com/www.amazon.com/index.html http://localhost/page_load_test/mobile_tp4/news.google.com/news.google.com/index.html http://localhost/page_load_test/mobile_tp4/m.news.google.com/news.google.com/index.html http://localhost/page_load_test/mobile_tp4/amazon.com/www.amazon.com/index.html http://localhost/page_load_test/mobile_tp4/m.amazon.com/www.amazon.com/index.html http://localhost/page_load_test/mobile_tp4/news.google.com/news.google.com/index.html http://localhost/page_load_test/mobile_tp4/m.news.google.com/news.google.com/index.html http://localhost/page_load_test/mobile_tp4/amazon.com/www.amazon.com/index.html http://localhost/page_load_test/mobile_tp4/m.amazon.com/www.amazon.com/index.html http://localhost/page_load_test/mobile_tp4/m.google.com/google.com/index.html http://localhost/page_load_test/mobile_tp4/m.accuweather.com/www.accuweather.com/index.html http://localhost/page_load_test/mobile_tp4/m.bbc.co.uk/www.bbc.co.uk/mobile/index.html/index.html http://localhost/page_load_test/mobile_tp4/m.nytimes.com/mobile.nytimes.com/index.html http://localhost/page_load_test/mobile_tp4/m.google.com/google.com/index.html http://localhost/page_load_test/mobile_tp4/m.accuweather.com/www.accuweather.com/index.html http://localhost/page_load_test/mobile_tp4/m.bbc.co.uk/www.bbc.co.uk/mobile/index.html/index.html http://localhost/page_load_test/mobile_tp4/m.nytimes.com/mobile.nytimes.com/index.html http://localhost/page_load_test/mobile_tp4/m.google.com/google.com/index.html http://localhost/page_load_test/mobile_tp4/m.accuweather.com/www.accuweather.com/index.html http://localhost/page_load_test/mobile_tp4/m.bbc.co.uk/www.bbc.co.uk/mobile/index.html/index.html http://localhost/page_load_test/mobile_tp4/m.nytimes.com/mobile.nytimes.com/index.html http://localhost/page_load_test/mobile_tp4/m.cnn.com/cnn.com/index.html http://localhost/page_load_test/mobile_tp4/m.twitter.com/twitter.com/toptweets/favorites.html/index.html http://localhost/page_load_test/mobile_tp4/m.wikipedia.com/en.m.wikipedia.org/index.html http://localhost/page_load_test/mobile_tp4/m.espn.com/m.espn.go.com/index.html http://localhost/page_load_test/mobile_tp4/m.cnn.com/cnn.com/index.html http://localhost/page_load_test/mobile_tp4/m.twitter.com/twitter.com/toptweets/favorites.html/index.html http://localhost/page_load_test/mobile_tp4/m.wikipedia.com/en.m.wikipedia.org/index.html http://localhost/page_load_test/mobile_tp4/m.espn.com/m.espn.go.com/index.html http://localhost/page_load_test/mobile_tp4/m.cnn.com/cnn.com/index.html http://localhost/page_load_test/mobile_tp4/m.twitter.com/twitter.com/toptweets/favorites.html/index.html http://localhost/page_load_test/mobile_tp4/m.wikipedia.com/en.m.wikipedia.org/index.html http://localhost/page_load_test/mobile_tp4/m.espn.com/m.espn.go.com/index.html
Test Results
Overall, it looks like the cache is no longer a loser on mobile, and it is, in fact, a winner! There were 2 ways in which the disk cache was tested, with the memory cache enabled at the same time, and with the disk cache being the only cache enabled.
Improvement From Enabling Disk Cache
| Galaxy Tab | Galaxy S | Sensation | Desktop | Desktop (1G) | |
| Disk Cache Only | 94%* | 8% | 16% | 39% | 45% |
| Both Caches | 14% | 15% | 13% | 11% | -6%** |
Notes: * There is obviously something wrong with this test - the numbers for having the disk cache enabled were in line with other devices, but the time taken when no cache was enabled was an order of magnitude larger. ** There has been speculation on why this entry is a loser. The going theory is that, since we don't write to memory cache with the disk cache enabled (except in a few minor circumstances), we're comparing "a bunch of hits in memory" to "a bunch of writes to disk followed by a bunch of hits", which would, indeed, perform worse. We should also note that Chrome has apparently noticed "performance issues" with a 1G disk cache (their theoretical limit), so in actuality they cap to around 300MB (and their cache is designed very similarly to ours).
Other Information
For some reason, the desktop took an order of magnitude longer than the mobile devices on every test. I attribute this to poor designing of the network conditions using NeckoNet (with NeckoNet turned off, the raw values were more in line with the values seen on the mobile devices).
On the Galaxy Tab and the Galaxy S, the cache was stored on device internal storage. However, it should be noted that both devices have notoriously bad filesystems for the partition on which the cache lived (the tablet appears to use some userspace filesystem implementation, while the phone uses Samsung's generally reviled RFS).
On the Sensation, the cache was stored on a 32GB class 4 MicroSD HC card. Differently-classed MicroSD cards could have some impact on the performance of these tests.
Improvements (Round 2)
For this round, memory cache was always disabled. One change I made for this run was to ensure that each device had all its tests run in quick succession (the previous test ran the tests for each cache size/combination on all devices before continuing to the next combination, allowing for changes in network conditions to possibly impact the results, with the exception of desktop, where the network was tightly controlled). Additionally, for this test, the Galaxy Tab was run on the Mozilla-G network instead of the higher performance Mozilla wireless network for better parity with the Galaxy S and the Sensation (as those 2 devices are only able to talk on the Mozilla-G network). Comparing with the numbers for the desktop above, we can see the improvements for mobile are much more in-line with desktop with this ordering of running the tests.
| Galaxy Tab | Galaxy S | Sensation | |
| 6MB Cache | 32% | 63% | 40% |
| 10MB Cache | 34% | 57% | 45% |
Next Steps
- Re-run the tests on the Galaxy Tab to get some (hopefully) more sensible numbers.
- Discuss with the mobile team what disk space usage is considered acceptable in both the "store internally" and the "store on sdcard" cases.
- Once we have this information, run these tests again with appropriately-sized caches.
- Discuss with the mobile team possible mitigation strategies for being killed with a dirty cache (possibly using the onPause event to sync disk cache activity and write _CACHE_MAP_ in a consistent state)
