Databases in AppScale

Hbase

HBase: Bigtable-like structured storage for Hadoop HDFS

Just as Google's  Bigtable leverages the distributed data storage provided by the  Google File System, HBase provides Bigtable-like capabilities on top of Hadoop Core. Data is organized into tables, rows and columns. An Iterator-like interface is available for scanning through a row range (and of course there is the ability to retrieve a column value for a specific key). Any particular column may have multiple versions for the same row key.

Performance and scalability.

In a web-driven world, datasets are larger than ever before – with “web scale” becoming the term of choice to describe the ultimate size of problems.

Hypertable is an open source project based on published best practices and our own experience in solving large-scale data-intensive tasks. Our goal is to bring the benefits of new levels of both performance and scale to many data-driven businesses who are currently limited by previous-generation platforms. Our goal is nothing less than that Hypertable become one of the world’s most massively parallel high performance database platforms.

Hypertable is a high performance distributed data storage system designed to support applications requiring maximum performance, scalability, and reliability.

Hypertable will be particularly invaluable to any organization that needs to manage rapidly evolving data to support demanding real-time applications. Modeled after Google's well known Bigtable project, Hypertable is designed to manage the storage and processing of information on a large cluster of commodity servers, providing resilience to machine and component failures. Hypertable seeks to set the open source standard for highly available, petabyte scale, database systems. 

http://incubator.apache.org/cassandra/ Video

Cassandra is a distributed storage system for managing structured data while providing reliability at a massive scale. 

Development of Cassandra started in Facebook in June 2007. It started of a system to solve the Inbox Search problem and since then has matured to solve various storage problems associated with structured/unstructured data.

Cassandra is a distributed storage system for managing structured data that is designed to scale to a very large size across many commodity servers, with no single point of failure. The philosophy behind the design of the storage portion of Cassandra is that it be able to satisfy the requirements of applications that demand storage of large amounts of structured data. Reliability at massive scale is a very big challenge. Outages in the service can have significant negative impact. Hence Cassandra aims to run on top of an infrastructure of hundreds of nodes (possibly spread across different datacenters). At this scale, small and large components fail continuously; the way Cassandra manages the persistent state in the face of these failures drives the reliability and scalability of the software systems relying on this service.

"Cassandra is written in Java and implements a sort of hybrid between Dynamo and Bigtable. (Both papers are worth reading.) It takes its distribution algorithm from Dynamo and its data model from Bigtable -- sort of the best of both worlds. Avinash Lakshman, Cassandra's architect, is one of the authors of the Dynamo paper." from Jonathan Ellis (http://spyced.blogspot.com/2009/03/why-i-like-cassandra.html)

SVN: http://svn.apache.org/repos/asf/incubator/cassandra/

.... Candidates

Project Voldemort

A distributed database 

http://project-voldemort.com/

Voldemort is a distributed key-value storage system

• Data is automatically replicated over multiple servers.
• Data is automatically partitioned so each server contains only a subset of the total data
• Server failure is handled transparently
• Pluggable serialization is supported to allow rich keys and values including lists and tuples with named fields, as well as to integrate with common serialization frameworks like Protocol Buffers, Thrift, and Java Serialization
• Data items are versioned to maximize data integrity in failure scenarios without compromising availability of the system
• Each node is independent of other nodes with no central point of failure or coordination
• Good single node performance: you can expect 10-20k operations per second depending on the machines, the network, and the replication factor
• Support for pluggable data placement strategies to support things like distribution across data centers that are geographical far apart.

It is used at LinkedIn for certain high-scalability storage problems where simple functional partitioning is not sufficient. It is still a new system which has rough edges, bad error messages, and probably plenty of uncaught bugs. Let us know if you find one of these, so we can fix it.

http://www.mongodb.org/display/DOCS/Home

Welcome to MongoDB

MongoDB is a high-performance, open source, schema-free document-orienteddata store that's easy to deploy, manage and use. It's network accessible, written in C++ and offers the following features :

• Collection oriented storage - easy storage of object-style data
• Full index support, including on inner objects
• Query profiling
• Replication and fail-over support
• Efficient storage of binary data including large objects (e.g. videos)
• Auto-sharding for cloud-level scalability (Q209)

High performance, scalability, and reasonable depth of functionality are the goals for the project.  Deep transaction support is not a goal of the system.

git://github.com/mongodb/mongo.git git://github.com/mongodb/mongo-python-driver.git The CouchDB Project http://couchdb.apache.org/index.html Apache CouchDB is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API. Among other features, it provides robust, incremental replication with bi-directional conflict detection and resolution, and is queryable and indexable using a table-oriented view engine with JavaScript acting as the default view definition language. CouchDB is written in Erlang, but can be easily accessed from any environment that provides means to make HTTP requests. There are a multitude of third-party client libraries that make this even easier for a variety of programming languages and environments. See the introduction and the technical overview for more information.