sccache - Google Code
http://code.google.com/p/sccache/

The SHOP.COM Cache System is an object cache system that...

SAVE N SHARE

database blog

A database lib

bdb alternative und sehr schnell

aid, here is what it takes to do 10.000 gets and sets:

Active record memory cache.

A Write-Through Cacheing Library for ActiveRecord

class Message < ActiveRecord::Base

InfoQ: Rockstar Memcaching http://www.infoq.com/presentations/lutke-rockstar-memcaching performance presentation

About the conference RubyFringe is an avant-garde conference for developers that are excited about emerging Ruby projects and technologies. They're mounting a unique and eccentric gathering of the people and projects that are driving things forward in our community.

Memcached tutorial video from the RubyFringe conference.

Video of memcached best-practices

RubyFringe presentation on Memcached

tobias lutke rubyfringe talk about memcached

CollegeHumor

CollegeHumor memcache use

Redis is a key-value database. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists and sets with atomic operations to push/pop elements.

“Redis is a key-value database. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists and sets with atomic operations to push/pop elements. “In order to be very fast but at the same time persistent the whole dataset is taken in memory and from time to time and/or when a number of changes to the dataset are performed it is written asynchronously on disk. You may lost the last few queries that is acceptable in many applications but it is as fast as an in memory DB (beta 6 of Redis includes initial support for master-slave replication in order to solve this problem by redundancy).”

A nice fast K/V data store, with some nice list/set features.

Plus a nice comment from Zed.

Traditional SQL databases with "ACID" properties (Atomicity, Consistency, Isolation and Durability) give strong guarantees about what happens when data is stored and retrieved. These guarantees make it easier for application developers, freeing them from thinking about exactly how the data is stored and indexed, or even which database is running. However, these guarantees come with a cost.

The first one is to use an in-memory data store, and communicate using the memcached protocol. This is, of course, *exactly* comparable to Memcached — behaviorally indistinguishable! — and it does worse. The second option is to do that, except switch to an on-disk data store. It’s pretty ridiculous that that’s still the same speed — communication overhead is completely dominating the time. Fortunately, Tyrant comes with a binary protocol. Using that substantially improves performance past Memcached levels, though less than a direct in-process database. Yes, communication across processes incurs overhead. No news here, I guess.

"Tokyo Tyrant is a server implemented on top of Cabinet that implements a similar key/value API except over sockets. It’s incredibly flexible; it was very easy to run it in several different configurations. The first one is to use an in-memory data store, and communicate using the memcached protocol. This is, of course, *exactly* comparable to Memcached — behaviorally indistinguishable! — and it does worse. The second option is to do that, except switch to an on-disk data store. It’s pretty ridiculous that that’s still the same speed — communication overhead is completely dominating the time. Fortunately, Tyrant comes with a binary protocol. Using that substantially improves performance past Memcached levels, though less than a direct in-process database. Yes, communication across processes incurs overhead. No news here, I guess."

scalablity

A software-based distributed caching system such as memcached is an important piece of today's largest Internet sites that support millions of concurrent users and deliver user-friendly response times. The distributed nature of memcached design transforms 1000s of servers into one large caching pool with gigabytes of memory per node. This blog entry explores single-instance memcached scalability for a few usage patterns.

"A software-based distributed caching system such as memcached is an important piece of today's largest Internet sites that support millions of concurrent users and deliver user-friendly response times. The distributed nature of memcached design transforms 1000s of servers into one large caching pool with gigabytes of memory per node. This blog entry explores single-instance memcached scalability for a few usage patterns."

I wanted a simple command-line tool to be able to grab real-time stats from memcache (memcached, I know, I know), and output it in a view something like top. I couldn't find anything like it, so I wrote one myself in perl. When writing it, I tried to keep it simple, portable, and lightweight. (No memcached perl modules required! I tried to keep it to modules I thought would be preinstalled on almost any modern system. It's also fairly polite - non-critical modules get checked, and if they aren't installed, the functionality is disabled without spewing errors or dying.) I realize it's not written well. But, hey, at least it exists, right? Until the day I released it, there wasn't any comparable tool like it for memcached. It gives you the basic stats, and not too much else. (You can specify thresholds, for instance, and it'll change color to red if you exceed the thresholds. You can also choose the refresh/ sleep time, and whether to show immediate (per second) stats, or lifetime stats.

Summary of the Facebook architecture and the bottlenecks they have had to work around

After considering a variety of data clustering algorithms, found that there was very little win for the additional complexity of clustering. So at Facebook, user data is randomly partitioned across indiviual databases and machines across the cluster. Hence, each user access requires retrieving data corresponding to user state spread across hundreds of machines. Intra-cluster network performance is hence critical to site performance. Facebook employs memcache to store the vast majority of user data in memory spread across thousands of machines in the cluster. In essence, nodes maintain a distributed hash table to determine the machine responsible for a particular users data. Hot data from MySQL is stored in the cache. The cache supports get/set/incr/decr and

Good overview of Django Caching Techniques

denormalization

"# LuxIO (ラックスIO)"# 普通のB+-tree # 特徴1 * mapped index * index部を全部mmap o index部を実メモリより小さいシステムが対象 # 特徴2 * 長いvalue * 4Gまで * node size(page size)をこえたvalueも余計なオーバーヘッドなしで扱える # 特徴3 * 効率的なappend * paddingなしでLinkedListのデータ構造 # SSDに向いてる？ # 使い道 * key-valともに小さいデータで構想なアクセスが必要な場合 * 実メモリ以下のデータベースという制約あり * 大きなvalueを扱いたい場合 * 大きなvalueをどんどん追記したい # 向かない処理 * 削除が多い処理 * 小さいデータをたくさんリンク o seekのオーバーヘッドが大きすぎる * Read,Writeの激しいアプリ # 分散はたぶんしない # Hashはつくるかも # read lockはなくしたい * 読み込みを重きをおく"

Key-Value型データ設計に関して。いくつかのシステムの特徴などのメモ。

"memcachedの特徴は、データをキャッシュするメモリーとして、通常のPCサーバーの物理メモリーを利用すること。大容量データを複数のPCサーバーのメモリーに分散しておくために、「キー・バリュー型データストア」と呼ぶ方法を採用している。データをいったん非正規化し、「キー」とそれに対応する「値（バリュー）」にしてから保存する。データをキーと値の組み合わせにすることで、複数のサーバーに分散しておける。"

a database implementing a dictionary, where every key is associated with a value. every single value has a type. The following types are supported: * Strings * Lists * Sets * Sorted Set (since version 1.1)

maybe the guy is not suitable to address such compare?

Persistent in-memory key value database compared to memcached

tructures and algorithms. Indeed both algorithms and data structures in Redis are properly choosed in order to obtain the best performance.

記者にとって驚きだったのは、現在日本で開発されているキー・バリュー型データストアがこの3つに留まらないことだった。しかも開発者は総じて若い。勉強会に参加する80人近くの技術者も、ほぼ同年代だった。

キー・バリュー型データストア（またはキー・バリュー型データベース）は、大量のユーザーとデータを抱え、データベースのパフォーマンス問題とコスト高に頭を悩ませるWeb企業が注目する技術である。

For those familiar with memcached, membase provides on-the-wire protocol compatibility, but adds disk persistence; hierarchical storage management; data replication; live cluster reconfiguration and rebalancing; and secure multi-tenancy with data partitioning. Like memcached, membase is simple, fast and elastic.

Persistent Key/Value Storage

Membase is an open-source (Apache 2.0 license) distributed, key-value database management system optimized for storing data behind interactive web applications. These applications must service many concurrent users; creating, storing, retrieving, aggregating, manipulating and presenting data in real-time. Supporting these requirements, membase processes data operations with quasi-deterministic low latency and high sustained throughput.

from oreilly news link

Membase is an open-source (Apache 2.0 license) distributed, key-value database management system optimized for storing data behind interactive web applications. These applications must service many concurrent users; creating, storing, retrieving, aggregating, manipulating and presenting data in real-time. Supporting these requirements, membase processes data operations with quasi-deterministic low latency and high sustained throughput. It scales linearly from a single-server deployment to a cluster of thousands of machines. And because membase does not require creation of a schema before storing data, it is a flexible, cost-effective place to Store Lots of Stuff.

Membase is an open-source (Apache 2.0 license) distributed, key-value database management system optimized for storing data behind interactive web applications. These applications must service many concurrent users; creating, storing, retrieving, aggregating, manipulating and presenting data in real-time. Supporting these requirements, membase processes data operations with quasi-deterministic low latency and high sustained throughput. It scales linearly from a single-server deployment to a cluster of thousands of machines. And because membase does not require creation of a schema before storing data, it is a flexible, cost-effective place to Store Lots of Stuff. The original membase source code was released as Open Source by NorthScale, Zynga and NHN to membase.org in June 2010.