Wednesday, March 26, 2008

The Open Source Google Stack

The open source community is slowly growing a software stack that emulates a number of internal Google technologies. What's interesting is that the stack is being developed by a number of tech companies -- ranging from giants (mostly Yahoo!) to medium size firms (Facebook) to startup companies (, PowerSet, Krugle, Veoh). So far, the following pieces of infrastructure have been developed

Hadoop is an umbrella project that covers a number of technologies: the Hadoop Filesystem (replacing Google's GFS), Map-Reduce, and HBase a database inspired by BigTable. In short, Hadoop covers storage technology as well as the infrastructure needed to compute over large amounts of storage.

Thrift is a RPC system which was initially developed by Facebook. While, in some senses, there is no lack of RPC-subsystems out there, Thrift is different. Thrift focuses on inter-language RPC. Interop between a large number of languages is transparent. Thrift also avoids the bloat that comes with some of the older RPC systems, such as SOAP. Finally, Thrift doubles as a serialization system (sort of like Python's Pickle or Java serialization). Because the binary data it generates is compact, it can be used for logging or in a database. Thrift is partially modeled off of Google's Protocol Buffers.

ZooKeeper is a service for coodination between distributed servers simplifying functions such as master election, configuration lookup, and distributed locks. ZooKeeper was recently open sourced by Yahoo.

These projects seem to be the start of a positive trend: a number of companies are realizing that in order to rapidly develop new services, they need to have infrastructure. In contrast to Google's typical strategy, these companies are sharing their work with each other. I think that this collaboration is critical to developing a robust infrastructure.

In some ways, I think that these developments put Google in a tough spot. It's not unforeseeable that these open source stack will grow to point where it is superior to Google's own stack. It would be unrealistic for Google to migrate to these new technologies. If there's going to be an open version of Google's technology, it would be in Google's best interest for that implementation to be their own. Not only would Google benefit from external improvements to their stack, it might make it easier for them to acquire new startups and integrate their technology into Google.