Webmaker/MakeDrive

MakeDrive

An offline-first, syncing filesystem for the web and Webmaker.

NOTE: The "MakeDrive" name is a working name only. Final naming and branding TBD

See this post by David Humphrey introducing MakeDrive.

Overview

Since its launch Webmaker has provided a set of tools (Thimble, Popcorn, Goggles) that focus on single-page makes (i.e., single file). Users have focused on creating content that is self-contained, without external dependencies other than URLs. This has been an effective way to insulate new web makers from the complexities of modern web development as they begin.

In 2014 Webmaker's tools are evolving to allow the creation and editing of more realistic, multi-file web sites, mobile web apps, components, etc. Today, everything on the web is a collection of separate resources--HTML, CSS, JavaScript, JSON, XML, images, and other types of media--some living on remote servers and used via URL, and some living on the same server, with relative links. Modern web making involves being able to work with trees of files and folders.

Adopting a files-and-folders approach makes it easier to bring traditional types of desktop editing tools to Webmaker, and gives users a well-understood metaphor. MakeDrive will be a web filesystem for web making, one that works online, offline, or mobile. Where other cloud storage services are a) integrated into the OS; b) focused on backup, video/photo sharing, etc. MakeDrive will be integrated into the web and web browser, and will focus specifically on the needs of web making.

MakeDrive will allow users to move easily between devices and browsers, and create and access their existing web makes. MakeDrive will synchronize user's files and folders between browsers and the cloud, and seek to provide novel and useful ways to put the user at the centre of their data. MakeDrive is also bound to makes.org. In the same way that gh-pages branches and github.io allow Github users to host static web sites, MakeDrive and makes.org allow Webmaker users to publish their makes easily.

Goals

MakeDrive has a number of near, medium, and long term goals. The following lists are incomplete, but point at the directions we might take.

Primary Goals

• MakeDrive is a web-first, user-first filesystem that tries embraces the open web
• A Webmaker user has a filesystem, available to various apps (e.g., Nimble, AppMaker).
• MakeDrive is a service that can be used in, and shared across various apps (i.e., it isn't part of Nimble, for example, but something Nimble uses)
• MakeDrive is a place to store web sites, web apps, and other web makes. It is optimized for HTML, CSS, JavaScript, images, JSON, and other web resources. It is not a backup service, a place to store all your photos, or share large files.
• MakeDrive expects transient users, and tries to make your files available wherever you go. You can always get your files on your current computer (i.e., in your current browser) and keep working.
• MakeDrive has no concept of file history, and never asks users to perform merges.
• MakeDrive uses a mix of an in-browser filesystem with cloud storage, and tries to handle syncing between the two without user intervention
• REST API with CORS for interacting with S3 cloud storage portion of filesystem

Secondary Goals

• MakeDrive can connect to other cloud systems, e.g., Dropbox, Github, GoogleDrive, etc in order to import/export data.
• Files can be public or private. Files saved in MakeDrive are potentially "published" and hosted on user.makes.org, but may not be (yet). Saving vs. publishing allows users to work on things privately.
• Easy to get files in/out of MakeDrive relative to current computer (drag-and-drop file upload)
• Simple sharing model, allowing a user to specify other users who can collaborate on a set of files
• Figure out the sweet spot between syncing everything always, and not hogging bandwidth, bloating browser storage
• Support for both trusted and untrusted computing environments. User might be using their laptop one day, and school computer the next.
• Develop a web filesystem conventional directory layout. Where a Unix filesystem has /bin, /etc/, /home, etc, what is the layout of MakeDrive?
  • /tmp - temporary files that are cleaned up on opening/closing the filesystem
  • /apps - folder where each app (e.g., /apps/nimble.webmaker.org, /apps/appmaker.webmaker.org) can place its files
  • /makes - user data (this name seems bad)
  • /docs - various kinds of learning and web editing docs for access offline
  • /script - scripts for things that can run on the filesystem (e.g., nohost web server)

Longer Term Research Goals

• Sync replaces Save as the dominant metaphor (consider https://twitter.com/gr2m/status/457921777822625792, http://branch.com/b/redesigning-the-save-symbol-let-s-do-this)
• Swappable, user-specified storage backends. We default to S3 for storage, but allow a user to specify their own backend, for example Dropbox.
• Special folders (virtual folders). For example: files by type (js, html, css), recently edited, trash, etc
• Special file types reflecting the fact that this is the web. For example, symlinks to URLs?
• Ability for users to obtain (or create--vanity urls) sharable URLs to files in the filesystem.
• Easy integration of MakeDrive in other web apps via Persona-like button(s). The user clicks the button and a File Picker is shown (e.g., https://www.dropbox.com/developers/dropins/saver, https://www.dropbox.com/developers/dropins/chooser/js)
• WebRTC or other LAN-based sync'ing for situations with no Internet but a local/mesh network
• Figure out how to support file syncing when there is no user account. For example, uuid generated that gets used to lookup a folder when re-connecting instead of a user account.
• Ability to upload files by sending an email (similar to how Kindle supports getting files on your Kindle via email attachments with a special email address per user)

User Stories

Examples of things we'd like MakeDrive to be able to support:

• MVP User Story: creating and editing an AppMaker component. Luke wants to create a Button component. He creates a new project, button-component and the following files:

component-button/
  component.html
  README.md
  style.css

He works until he's happy, and saves his work. Later he goes to AppMaker and is able to try his component using the following URL:

https://drive.webmaker.org/components/component.html

This allows Luke (and only Luke) to preview his new component in AppMaker. Later, after editing and fixing bugs, he decides to publish his component. He now shares his work via makes.org:

https://luke.makes.org/components/component.html

He later finds a bug and fixes it, and his public component has the new fix.

MVP #1 Storyboard

• MVP User Story Version 2 - Remixing a component from within Appmaker. Luke is using the stock Appmaker button component, but wants to change it. He remixes it, which spawns a Nimble editor and creates a copy of the Button component in his Make Drive. He can made changes to the remixed button component which are reflected in Appmaker.

MVP #2 Storyboard

• Jen has never used Webmaker before. She visits nimble.webmaker.org and sees an editor with an index.html page. She is able to make changes and save them. After editing this page for a while, she decides to publish it, and claims jen.makes.org, where her new page is now visible.

• A few days later, Jen decides to add an image to her home page on jen.makes.org. She goes back to nimble.webmaker.org, and is able to open her existing work. She drags-and-drops the image into the editor, and it gets "uploaded".

• At work the next day, Jen sees an animated gif that she wants to add to her page. She uses her work computer to visit nimble.webmaker.org. She is able to sign-in and her files appear, where she adds the gif, saves, and publishes, before sharing the link to her jen.makes.org/ page with her colleague.

• Claudia has created many web components for AppMaker using Nimble. While attending a hackathon, she decides to create a new one by using one of her old ones as a template. She doesn't have Internet access at the event, but is able to open nimble.webmaker.org, find her old component, and save it with a new name, before editing it. She uses Nimble's built-in web server (nohost) to preview and demo her work at the event.

• Wei is traveling and has an afternoon to kill. He visits a local library, where he has a great idea for a web page that shows a map of where he's been along with some photos. He visits nimble.webmaker.org, and specifies that he's using an untrusted computer. He is able to access his old files, where some of his images live. He creates a new web page, uses a Map API in JS, saves and publishes everything before emailing his friends the link to his make. When he's done he closes the browser he's been using, which deletes all the files he was accessing locally (they are saved in Webmaker's server).

• Hope is at her brother's house. He's working on a project for school, and struggling to get a piece of JavaScript to work. Hope knows she's done something similar in the past, and pulls out her phone. She's able to view her old Webmaker files using MakeDrive, and finds exactly what she wants. She shares a URL with her brother, and he's able to open her JS file, and they use it to solve his problem.

Technology

• S3 is our backend
• MakeDrive is a few things:
  • a node.js backend to proxy S3 requests using Webmaker auth. NOTE: MakeDrive does not (currently) deal with login, and assumes that users are logging in using Webmaker apps on another subdomain of *.webmaker.org (i.e., it uses a shared super cookie via the webmaker-auth npm module).
  • a way for logged in users to see their files via URLs (e.g., from AppMaker they can see their saved work in MakeDrive)
  • a client-side library that mixes Filer, node/S3 communication, Webmaker auth, sync
• MakeDrive will integrate with makes.org for publishing. The details of this are still being worked out, but probably it will mean copying data between S3 buckets, and allowing a URL like user.makes.org/folder/file to do what you'd expect and serve /folder/file from the MakeDrive. Longer term this probably means that user.makes.org/index.html is the first file you see/edit in your MakeDrive.
• We need to figure out how to connect MakeDrive to the engagement/metrics workflow, especially so that we can watch for issues like rising S3 bills.
• The node/S3 proxy can use S3's API, and probably we can reuse an existing node.js module that does a bunch of this (knox?)
• Need to figure out push notifications so that S3 proxy can let local filesystem know when things are saved or changed. Probably EventSource (polyfill for IE which doesn't support) to do server-side events.
• On the client side we have to sort out things like rsync across the network divide (sockets? xhr?), VFS in Filer so you we can combine local and remote filesystems, and other dependent work.

MakeDrive's Sync UX

With MakeDrive we have a chance to do something different, offer something familiar, something useful, and something of the web.

I want the web to win because it is going to enable rich always-up-to-date, safe, frictionless, easy-to-share, instant-on, multi-screen experiences. All of these are features targeted at users, and none of them possible in a native world.--from http://paulbakaus.com/2014/04/16/winning-for-users/

There are lots of cloud-based storage systems, and we can learn from them. None of them is specifically designed for the web, all of them assuming an OS and native filesystem. Often, they work best when their design is so carefully folded into the OS that you don't have to learn how to use them (e.g., the /Dropbox folder).

MakeDrive is also a cloud-based storage system, but with a browser and a browser-filesystem replacing the OS + native filesystem. This design has some drawbacks:

• the browser filesystem can't be merged into the native filesystem. It's possible to get files in and out of the browser and native filesystems, but it requires careful design work
• the user is likely going to struggle with the idea of a "filesystem in their browser," and so it has to be carefully disguised so as not to confuse
• the browser filesystem is tied to a particular browser

This design has some advantages:

• you don't have to install anything to create a new client--just visit a web page, and it happens automatically
• you can get your filesystem on any computer or device that has a web browser, including mobile devices, where creating your own sync apps can be greatly restricted by App Stores
• you can move your files (sync) across devices, operating systems, etc. The browser, and the web, are the platform

Designing an always-available, always sync'ing web filesystem isn't as simple as copying what's been done before. Browser users are different from desktop/OS users. For example, a desktop storage client can run at startup and then stay active all the time in the background. A browser storage client has to deal with the transient nature of the user, who is unlikely to spend hours on the same thing. Moving data from the cloud to the local/browser filesystem is still the same problem (i.e., you have to more the same number of bits either way).

The problem can be described as the problem of connecting Here and There. Here is where we are, where we are working right now. This Here is unstable, and may move from computer to computer, device to device. We can better think of Here as This Here since it doesn't have the same stability as There. There is always elsewhere, but may or may not be accessible from This Here (e.g., no Internet), and it's important to note that we may choose this situation, or it may be chosen for us. In other words, there is sometimes a way from This Here to There, but not always.

The relationship from This Here to There is less interesting to users than to us. As a user, I am always Here. I may want to work with a particular file or set of files, and they might be There or available in This Here, but neither matters to me: I want them Here. This requires one of a number of things to happen:

• Some automatic system needs to sync files between This Here and There such that I can satisfy myself with only ever being Here; or
• I need to be made aware of what is available There vs. in This Here and be able to choose what I sync between the two.

There are tradeoffs. We need to think about the size of data we'll need to share over the current connection. We also need to decide how much we want to decide this for the user. On the one hand, it seems like access to everything is better, always. But in the case of an untrusted computer, I want as little to get sync'ed as possible--just enough to work right now ("Open my folder for Project X but nothing more").

The design problem is one that needs to use familiar UI, but augment it in ways that allow us to solve for our problems above. That is, we have dialog boxes for Open, Save As..., trees of files and folders, etc. and we have to add to these in ways that let us do what we need. Dropbox achieves this with extra icon overlays (green checkmarks, animated syncing arrows) to show the status of a file/folder.

Questions

• How is MakeDrive different from X cloud storage? Which other services exist in this space?
  • Github (source code, advanced users/developers)
  • Micosoft OneDrive (Windows, photos)
  • Dropbox (Any OS, sharing with friends, photos)
  • Google Drive (ChromeOS, Google Docs, Spreadsheet, etc. Apps)
  • Box (IT and Business users, focus on privacy settings)
  • Amazon Cloud Drive (Amazon Fire, photos/video, backup)
  • Bittorrent Sync (large files, p2p)

Where most cloud platforms are increasingly targeting business (Box, S3) or personal (Dropbox, Google Drive, etc focused on video, photo, document backup) use cases, MakeDrive is focused on files, and collections of files for creating things on the web. MakeDrive attempts to seamlessly bridge offline and online in-browser editing/making. MakeDrive allows Webmaker users to collaborate and share things they make

• What is our file size restriction (max size per file)?
• How much total storage do we allow?
• What does sharing look like? Read-only, collaborator (can get Read+Write), owner?
• Should we store our apps in the filesystem?

Resources

• MakeDrive Github Repo
• Filer - browser based filesystem
• nohost - browser based web server
• Nimble