Data Store API

This is the snapshot of a discussion between Mounir Lamouri, Thinker Lee, Gene Lian, Jonas Sicking and Hsin-Yi Tsai.
The work drafting happened on https://etherpad.mozilla.org/whatever-you-want

Use Cases

Allow an application to create data that can be shared with multiple other applications.
Allow multiple applications supply data to the same data store.
Support read-only stores like facebook contacts.
Support read/write stores like built-in contacts.
Support keeping an application-local cache of a data store. I.e. enable getting notified about changes to a data store so that the local cache can be kept up-to-date.

Why not...?

Why not use specific APIs for the use-cases like the existing Contacts and SMS/MMS APIs?

Here's an informative reply from Jonas Sicking on the dev-webapi list, included below:

The Contacts and MobileMessage APIs are richer in the sense that they support things like richer querying, like filtering, sorting and grouping.

However the Contacts and MobileMessage APIs has the severe shortcoming is that you are forced to live with the limitations of what querying capabilities those APIs have. Including the performance of those quering API.

We are *constantly* having to revise these APIs because it turns out that the querying capabilities aren't matching what our apps need. This is not a workable long term situation. And it's not even a workable short-term solution for 3rd party apps since we can't revise the APIs to support the capabilities that every 3rd party app developer needs.

This is why the DataStore API allows applications to synchronize data into a application-local cache. This cache can be stored/index/grouped/sorted in whatever format the application needs in order to support its UI. It even allows things like merge data from the MobilaMessage API and the Contacts API into a single location.

A very common reaction to this is "caching data in the application means duplicating the data!". This is true, but the current path we're on also causes a lot of duplication. Supporting all types of filtering, sorting and grouping like we do, forces us to create a lot of indexes. Indexes are effectively partial copies of the data. And we have to create those indexes whether they are actually used or not, because the API requires them.

By allowing applications to cache the data, only data that is actively needed by installed application is copied. And we enable applications to cache data in more formats than we could every think of and bake into the API.

This obviously doesn't mean that DataStore API will solve all of our problems. I suspect that something like the inter-app communication API will still be needed.

Why not use the Inter App Communication API?

WebAPI/Inter App Communication has more complicated security issues and is being worked at a lower priority.

Interface

 typedef (DOMString or unsigned long) DataStoreKey;
 
 interface DataStore : EventTarget {
   // Returns the label of the DataSource.
   readonly attribute DOMString name;
 
   // Returns the origin of the DataSource (e.g., 'facebook.com').
   // This value is the manifest URL of the owner app.
   readonly attribute DOMString owner;
   
   // is readOnly a F(current_app, datastore) function? yes
   readonly attribute boolean readOnly;
 
   // Promise<any>
   Promise get(DataStoreKey... id);
     
   // Promise<void>
   Promise put(any obj, DataStoreKey id, optional DOMString revisionId = "");
  
   // Promise<DataStoreKey>
   Promise add(any obj, optional DataStoreKey id, optional DOMString revisionId = "");
 
   // Promise<boolean>
   Promise remove(DataStoreKey id, optional DOMString revisionId = "");
 
   // Promise<void>
   Promise clear(optional DOMString revisionId = "");
 
   readonly attribute DOMString revisionId;
   
   attribute EventHandler onchange;
   
   // Promise<unsigned long>
   Promise getLength();
   
   DataStoreCursor sync(optional DOMString revisionId = "");
 };
 
 interface DataStoreCursor {
 
   // the DataStore
   readonly attribute DataStore store;
 
   // Promise<DataStoreTask>
   Promise next();
 
   void close();
 };
 
 enum DataStoreOperation {
   "add",
   "update",
   "remove",
   "clear",
   "done"
 };
 
 dictionary DataStoreTask {
   DOMString revisionId;  
 
   DataStoreOperation operation;
   DataStoreKey id;
   any data;
 };
 
 dictionary DataStoreChangeEventInit : EventInit {
   DOMString revisionId = "";
   DataStoreKey id = 0;
   DOMString operation = "";
   DOMStirng owner = "";
 };
 
 [Constructor(DOMString type, optional DataStoreChangeEventInit eventInitDict)]
 interface DataStoreChangeEvent : Event {
   readonly attribute DOMString revisionId;
   readonly attribute DataStoreKey id;
   readonly attribute DOMString operation;
   readonly attribute DOMString owner;
 };

Manifest

For the application that provides the datastore

 {
   ...
   datastores-owned: {
     "contacts": {
       "access": "readonly",
       "description": ...
     }
   },
   ...
 }

For the application that wants to access the datastore

 {
   ...
   datastores-access: {
     "contacts": {
       "readonly": true,
       "description": "Facebook contacts",
     }
   },
   ...
 }

Revisions and changes

The revisionId is a UUID and it can be used to retrieve the delta between a particular revisionId and the current one using |sync()|

Examples

Basic operations

  // Here we retrieve the list of DataStores called 'contacts'.
  navigator.getDataStores("contacts").then(function(stores) {
    dump("DataStores called 'contacts': " + stores.length + "\n");
    
    if (!stores.length) return;
    
    dump("Current revisionID: " + stores[0].revisionId + "\n");
    
    // Retrieve an object from the first DataStore.
    stores[0].get(42).then(function(obj) {
      // ...
      
      // Update an object
      obj.nick = 'baku';
      stores[0].put(obj, 42).then(function(id) {
        // id == 42
        // ...
      }, function(error) {
        // something wrong happened. Error is a DOMError object.
      });
    });
    
    // Delete an object
    stores[0].remove(23).then(function(success) {
      if (success) {
        // The object has been deleted.
      } else {
        // Object 23 didn't exist.
      }
    });
    
    // Storing a new object
    stores[0].add({ "nick": "baku", "email" : "a@b.c" }).then(function(id) {
      // ...
    });
  });

Sync

The synchronization of a DataStore with a 'private' app storage can be done using the 'sync' method. Calling this method, DataStore creates a DataStoreCursor that helps the app with the synchronization starting from scratch or for a valid revisionId. The sync operation can be terminated calling cursor.close(). If something changes in the DataStore when the cursor is synchronize the app, all the changes will be managed by the cursor as additional operation: this means that when the cursor completes its tasks, the app will be always in sync with the current revisionId of the DataStore.

The basic usage of the cursor is this:

 navigator.getDataStores('contacts').then(functions(stores) {
   if (!stores.length) return;
 
   let cursor = stores[0].sync(/* a revisionId can be used here. If it's invalid it'll be ignored */);
 
   function cursorResolve(task) {
     // task.operation describes what the app has to do in order to be in sync with the current revision of this datastore.
     switch (task.operation) {
        case 'done':
          // No additional operation has to be done.
          dump("The current revision ID is: " + task.revisionId + "\n");
          return;
 
       case 'clear':
         // All the data you have are out-of-sync. Delete all of them.
         break;
 
       case 'add':
         // A new object has to be inserted
         dump("Adding id: " + task.id + " data: " + task.data + "\n");
         break;
 
       case 'update':
         // Something has to be updated
         dump("Updating id: " + task.id + " data: " + task.data + "\n");
         break;
 
       case 'remove':
         dump("Record: " + task.id + " must be removed\n");
         break;
     }
 
     cursor.next().then(cursorResolve);
   }
 
   // Cursor.next() always returns a promise.
   cursor.next().then(cursorResolve);
 });

Issues

* {name, owner, value} is a complicated key.
* UI: what to do when we have multiple access requests?
* Should all data stores with the same name share a schema?
* Enforcing types can be a footgun. What should a data provider do if it decides some key should have a different type?