Research Abstracts - 2007

Motivation

Distributed applications commonly require the sharing of storage between their components. Today's systems rely on specialized storage and data transfer mechanisms: some cooperative web caches use DHT techniques to replicate data, PlanetLab experiments use scp to centrally collect results, and Grid applications have customized file systems that position data close to computation. This research explores whether it is possible and desirable to build a distributed file system that can serve as the storage component for many wide-area applications.

Designers of distributed applications could simplify their lives by delegating many storage concerns to a distributed file system. A single filesystem name space visible from all nodes would save each application from having to provide its own naming and data lookup scheme. Similarly useful would be infrastructure support for fault-tolerance, efficient wide-area data location and transfer, and control over consistency and failure semantics.

Are there storage requirements that are common across many distributed applications? Can these requirements be met by a single general-purpose file system design? We believe the answer is yes, but that existing distributed file systems, such as AFS, NFS, and SFS, are not sufficient. Our experience with SFS on the PlanetLab and RON testbeds suggests two reasons.

First, most popular distributed file systems stress transparency: a remote file should behave just like one on a local-disk file system. This forces the file system to implement expensive strict consistency semantics, as well as to try hard (via long timeouts) to mask temporary server failures. Many wide-area applications don't need these semantics; for example, a cooperative web cache would rather re-fetch from the origin server than wait for the file system to find the latest copy of a cached page.

Second, widely-used distributed file systems typically store entire subtrees (or the whole file system) at a single site. This property is awkward when output generated at one site is input at another; sending data through a central server creates a needless bottleneck. Centralization is also a problem for applications that are distributed to increase availability.

The need for a global file system has been articulated for PlanetLab (Section 3 of [1]) and GENI (see the distributed services document on the GENI web site), and experience with Grids also suggests that such a file system would be valuable [2]. Though it operates at a cluster level, Google's GFS [3] suggests that a shared distributed storage infrastructure can benefit many different applications.

Research Agenda

This research focuses on the design and implementation of WheelFS [4], a global file system with convenient location-independent names that gives applications control over data placement, failure behavior, and consistency. WheelFS incorporates the following new ideas. First, it augments the POSIX interface with semantic cues that allow applications to select desired behavior in the presence of failures or inconsistent updates. Second, it implements a policy of writing locally and reading globally, so that outputs can be written quickly and inputs can be fetched directly from the copy that is nearest on the Internet. Finally, the design relaxes certain POSIX semantics in a way that we believe will harm few practical applications yet allow the system to provide good performance. It is not intended to store users' home directories, due to these weakened semantics and a focus on performance and flexibility rather than durability; instead, we intend WheelFS to function as the storage component of large-scale distributed applications, allowing users to concentrate on their application's details and semantics, rather than data storage issues.

This research also explores redesigning and/or instrumenting many existing applications to use WheelFS, and evaluating the design and performance impact of WheelFS on those applications. Possible candidate applications for this purpose include a distributed measurement infrastructure, a cooperative web cache, a distributed digital library, and a Grid computation infrastructure. WheelFS should greatly simplify the implementation of these applications, hopefully with minimal performance impact (and perhaps some performance improvement).

References:

[1] Thomas Anderson and Timothy Roscoe. Learning from PlanetLab. In The Proceedings of the 3rd WORLDS, November 2006.

[2] George Kola, Tevfik Kosar, Jaime Frey, Miron Livny, Robert J. Brunner and Michael Remijan. DISC: A System for Distributed Data Intensive Scientific Computing. In The Proceedings of the 1st WORLDS, November 2004.

[3] Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung. The Google File System. In The Proceedings of the 19th SOSP, October 2003.

[4] Jeremy Stribling, Emil Sit, M. Frans Kaashoek, Jinyang Li and Robert Morris. Don't Give Up on Distributed File Systems. In The Proceedings of the 6th IPTPS, February 2007.