Networked Persistence

In a comment, Neel Krishnaswami asked us to discuss networked persistence. Here are some preliminary thoughts on this.

The most important thing about persistence in any context is that it should fit naturally with the language. This means lots of things: internal-external data interchange; persistent storage identification and management; hooking up to the language's control model; and more. For now I'll discuss these three points. Notice that the questions themselves aren't really about networked persistence, but the network will show up in each of the answers.

Data interchange on the Internet naturally demands an “XML story”, and lots of language research is trying to wrestle with this. If you step outside research and spend some time in blog-space or studying Web service APIs, you see a very interesting trend: the growing acceptance and even promotion of JSON. Because JSON obviously integrates nicely into JavaScript, and its support is increasing, we're ducking the XML question entirely for now. In fact, our Web services API automatically converts XML data into JSON for your convenience.

What is persistent storage? On a local system, computer scientists find it useful to categorize at least three different kinds of storage: the database, the persistent heap, and the filesystem. (And that's arguably just a 1970s view of systems.) We can distinguish between these along several dimensions: the object model, the power of query, the style of naming, the interaction with processes, etc. In Flapjax, though, many of these dimensions lose their relevance for two important reasons. First, we don't have (and don't want) a true distributed operating system, so the “process” boundary dies at the network interface (with end-to-end services taking its place). Second, network latency greatly distorts the cost models. So some unification of these concepts may be possible. For now, Flapjax gives each user a “home object”; their data are held by fields. From the program's viewpoint it's just a persistent object, but the user can think of fields as a pun for subdirectories, and use the object-browser like a file-browser. What we definitely don't have is a story on search, aggregation, and other functions you get from SQL. We have various thoughts on this, ranging from leveraging XPath to letting users define schemas, but (indeed, therefore!) this is very much an open question.

The control front is somewhat obvious: because Flapjax computations are time-varying, changes must naturally be pushed to the server, and server changes must naturally trigger renewed computation on the client. Defining this crisply has proven to be somewhat tricky. One issue is handling aggregate data, which Michael Greenberg is studying from first principles. The other is how to deal with multiple clients that write simulaneously. Obviously there is already research on this topic, but we do have our hands tied by the lack of server-push, the sheer number of clients that may be accessing a datum, existing APIs, etc. We have our own distributed algorithm that helps with this, but just defining “reasonable” behavior is tricky: if you don't get it right, a client who is sharing a writeable buffer will see the new characters they type disappear before their very eyes.

We need to clean things up a little more before we say much. My reticence isn't in the usual academic “I have a paper coming up and I don't want someone to scoop it or find the flaw in it before it gets in” manner—rather, we really do have more design work to.

And, of course, we welcome feedback and prioritization.