DOM/QuotaManager/SchemaChanges

From MozillaWiki
< DOM
Jump to: navigation, search

This page tracks schema changes to Firefox storage subsystems that may cause downgrade problems. It's primarily concerned with QuotaManager and its clients, but other subsystems are listed here as well.

Testing page: https://firefox-storage-test.glitch.me/

Last updated: May 15, 2018 with Firefox 62 on mozilla-central trunk.

Schema Changes By Release Version

Pending: Currently targeted for Firefox 62 https://bugzilla.mozilla.org/show_bug.cgi?id=1286798 Next generation LocalStorage

  • QuotaManager and LocalStorage will be affected, but is currently slated to be downgrade friendly.

Firefox 62 https://bugzilla.mozilla.org/show_bug.cgi?id=1239708

  • Places v48

Firefox 61 https://bugzilla.mozilla.org/show_bug.cgi?id=1452660

  • Places v47

https://bugzilla.mozilla.org/show_bug.cgi?id=1293445

  • Places v46

https://bugzilla.mozilla.org/show_bug.cgi?id=1426554

  • Places v45

https://bugzilla.mozilla.org/show_bug.cgi?id=1310295

  • Places v44

Firefox 60 https://bugzilla.mozilla.org/show_bug.cgi?id=1313188

  • Places v43

https://bugzilla.mozilla.org/show_bug.cgi?id=1346554

  • Places v42

Firefox 59 Firefox 58 https://bugzilla.mozilla.org/show_bug.cgi?id=1356531

  • Places v41

https://bugzilla.mozilla.org/show_bug.cgi?id=1399879

  • Places v40 (soft failures - Places bumps schema version up/down)

Firefox 57 https://bugzilla.mozilla.org/show_bug.cgi?id=1404344

  • MAJOR HACKY DOWNGRADE FIX.
  • Quota Manager v3 aliased back down to v2.1
  • Cache API v27 introduced that masquerades as v25 on disk
  • (ServiceWorkerRegistrar changes left intact because that does not result in hard breakage.)

https://bugzilla.mozilla.org/show_bug.cgi?id=1290481

  • Quota Manager v3 (storage.sqlite)
  • Cache API v26

https://bugzilla.mozilla.org/show_bug.cgi?id=1353636

  • ServiceWorkerRegistrar v8

https://bugzilla.mozilla.org/show_bug.cgi?id=1396282

  • Places v39 (soft failures)

Firefox 56 https://bugzilla.mozilla.org/show_bug.cgi?id=1352502

  • Places v38 (soft failures)

Firefox 55 https://bugzilla.mozilla.org/show_bug.cgi?id=1339081 (primary), https://bugzilla.mozilla.org/show_bug.cgi?id=1314361, https://bugzilla.mozilla.org/show_bug.cgi?id=1311057

https://bugzilla.mozilla.org/show_bug.cgi?id=1251238

  • ServiceWorkerRegistrar v7

https://bugzilla.mozilla.org/show_bug.cgi?id=977177

  • Places v37 (soft failures)

Firefox 54 https://bugzilla.mozilla.org/show_bug.cgi?id=1322111

  • Cache API v25

Firefox 53 https://bugzilla.mozilla.org/show_bug.cgi?id=1290944

  • ServiceWorkerRegistrar v6 (not QM, handles downgrades by clearing registry but does not break)

https://bugzilla.mozilla.org/show_bug.cgi?id=1181127

  • ServiceWorkerRegistrar v5 (not QM, handles downgrades by clearing registry but does not break)

https://bugzilla.mozilla.org/show_bug.cgi?id=1258127

  • Places v36 (soft failures)

Firefox 52 https://bugzilla.mozilla.org/show_bug.cgi?id=1264178

  • Cache API v24

https://bugzilla.mozilla.org/show_bug.cgi?id=768074

  • Cache API v23

https://bugzilla.mozilla.org/show_bug.cgi?id=964561 (store large structured clones as separate files)

https://bugzilla.mozilla.org/show_bug.cgi?id=1302901

  • Places v35 (soft failures)

Firefox 51 https://bugzilla.mozilla.org/show_bug.cgi?id=1302036 (via aurora uplift, central was 52) (2016-10-29)

  • IndexedDB v25
  • JS Structured Clone v8

https://bugzilla.mozilla.org/show_bug.cgi?id=1187335 (2016-09-08)

  • Cache API v22

Firefox 50 https://bugzilla.mozilla.org/show_bug.cgi?id=1264053 (via aurora uplift, central was 51)

  • IndexedDB v24
  • JS Structured Clone v7

Firefox 49 https://bugzilla.mozilla.org/show_bug.cgi?id=1195930 (2016-06-06)

  • Quota Manager v1 (storage.sqlite, also introduced .metadata-v2)
  • IndexedDB v23

https://bugzilla.mozilla.org/show_bug.cgi?id=1243792 (2016-05-10)

  • Cache API v21

(see "by subsystem" for older versions)
Related Changes By Release Version

Firefox 53 https://bugzilla.mozilla.org/show_bug.cgi?id=1321550 nsPrincipal::GetOrigin now truncates query and hash parts of the origin for file: URI's. This did not impact schemes other than "file" (and maybe "chrome:///blah"?), because:

  • "about", "moz-safe-about", "indxeddb" specs all fast-path out and use GetAsciiSpec.
  • non-"chrome" protocols which return something to GetAsciiHostPort(). This does not cover file URI. It seems like "chrome:///" and "resource:///" may have been impacted if they use a ContentPrincipal, but neither of those should successfully load as a navigation with a content principal.

https://bugzilla.mozilla.org/show_bug.cgi?id=1286914

  • QM learned to support the "moz-extension" scheme.

Firefox 50 https://bugzilla.mozilla.org/show_bug.cgi?id=1280336

  • QM learned to support the "about" scheme. Previously it only understood "moz-safe-about".

Firefox 49 https://bugzilla.mozilla.org/show_bug.cgi?id=1195930

  • QM learned to support the "resource" scheme.


By Subsystem (alphabetical) and how it handles failures:
"Hard failures" mean that the subsystem checks the schema version and fails to initialize or work if downgraded. "Soft failures" mean that the subsystem will try and use the database but may encounter failures. "Schema down-rev'ed" means that the schema version is set to the schema version the current version of Firefox understands, which means that the migration logic will be re-run if/when the profile is used on a more modern Firefox release.


Cache API (QuotaManager client). Hard failures on downgrade.

IndexedDB (QuotaManager client). Hard failures on downgrade. v26 (concurrent with QM v2, LocalStorage v2), Firefox 55, https://bugzilla.mozilla.org/show_bug.cgi?id=1339081 v25 (concurrent with Structured Clone v8), Firefox 51 (via aurora uplift, central was 52), https://bugzilla.mozilla.org/show_bug.cgi?id=1302036

v24 (concurrent with Structured Clone v7), Firefox 50 (via aurora uplift, central was 51), https://bugzilla.mozilla.org/show_bug.cgi?id=1264053 v23 (concurrent with QM v1), Firefox 49, https://bugzilla.mozilla.org/show_bug.cgi?id=1195930 v22 (concurrent with Structured Clone v6), Firefox 44, https://bugzilla.mozilla.org/show_bug.cgi?id=1201620 v21, Firefox 43, https://bugzilla.mozilla.org/show_bug.cgi?id=1202788 v20, Firefox 43, https://bugzilla.mozilla.org/show_bug.cgi?id=771288 v19, Firefox 43, https://bugzilla.mozilla.org/show_bug.cgi?id=871846 v18, Firefox 40, https://bugzilla.mozilla.org/show_bug.cgi?id=1131776 JS Structured Clone (JS subsystem used by IndexedDB persistence). Hard failures on downgrade.

LocalStorage/DOMStorage (Independent, soon to be a QuotaManager client).

Places (Never going to be a quota manager client). Soft failures on downgrade, schema down-rev'ed.

Push (preliminary investigation suggests hard DB breakage on downgrade, but there may be an error handling path I'm missing: 10/17/2017) (each push service has its own database with its own revision, although they track the underlying IDB schema) pushAndroidGCM

pushHttp2

pushapi (websocket)

Quota Manager (aka QM). Hard failures on downgrade, including hard failures for all

ServiceWorkerRegistrar: (Not currently a quota manager client, although the Cache API is and that's also essential for ServiceWorkers to operate.) Currently handles downgrades by deleting PROFILE/serviceworker.txt, effectively removing all ServiceWorkers and orphaning/leaking their storage (DOM Cache API currently has no reconciliation between used chrome storage and registrations).


Error Reporting (when things go wrong, what hints do we have?):

Quota Manager: QM_WARNING reports the specific file and line where an error occurred to the console service using a classification of "quota". (This happens via mozilla::dom::quota::ReportInternalError and nsContentUtils::LogSimpleConsoleError.) NS_WARNING will also be invoked with the generated string, which will be logged to stderr/logcat ("Gecko"). This general mechanism has been around since https://bugzilla.mozilla.org/show_bug.cgi?id=1124189 landed in Firefox 38. Prior to this landing, there were a number of nsPrintfCString's used to build a message that was logged via NS_WARNING,

  • "Origin '%s' failed to parse, handled tokens: %s" reported during origin parsing which will occur during origin initialization during QM's initialization phase at startup. (It may also occur during an attempt to create a new origin #ifndef RELEASE_OR_BETA, see the "Preventing creation..." error below.) Around since https://bugzilla.mozilla.org/show_bug.cgi?id=1124189 landed in Firefox 38.
  • "Preventing creation of a new origin directory which is not supported by our origin parser or is obsolete!": #ifndef RELEASE_OR_BETA in Firefox 53 as part of https://bugzilla.mozilla.org/show_bug.cgi?id=1286914
  • There are a number of other origin parsing errors that may happen too.
  • UNKNOWN_FILE_WARNING(_leafName) sub-type, "Something (%s) in the directory that doesn't belong!" reported during origin initialization and/or upgrades. New in Firefox 55 as part of https://bugzilla.mozilla.org/show_bug.cgi?id=1339081.

IndexedDB: IDB_WARNING pre-dates QM_WARNING and is effectively identical to it, the only difference being it logs to the console with a classification of "indexedDB". It has existed since https://bugzilla.mozilla.org/show_bug.cgi?id=958359 landed in Firefox 30, although it only gained sprintf powers in https://bugzilla.mozilla.org/show_bug.cgi?id=1339826 in Firefox 55. There is also an IDB_REPORT_INTERNAL_ERR variant without the NS_WARNING, plus an IDB_ENSURE_TRUE and IDB_ENSURE_SUCCESS that build on IDB_REPORT_INTERNAL_ERR.


To improve:


Upcoming Changes
The following bugs are expected to result in schema changes:


Current Versions
The list of current subsystem versions with searchfox queries to make it easier to tell when things have changed. This duplicates the "By Subsystem" info up above but makes it easier to read.


Cache API

IndexedDB

JS Structured Clone

LocalStorage

Places

Push pushAndroidGCM:

pushHttp2

pushapi (websocket)

QuotaManager

ServiceWorkerRegistrar



Retrieved from "https://wiki.mozilla.org/index.php?title=DOM/QuotaManager/SchemaChanges&oldid=1194384"