Wednesday, October 28, 2009

My (not so) secret plan

Did some work on re-packing Cassandra as an OSGi based daemon. The secret plan is to have 3 eclipse products :

Cassandra node


Cassandra is the clustered DB. This kind of databases are designed with high availability in mind and operates in "multi-master" mode by default. Optimal configuration starts at 3 nodes : set your database to "validate transactions" when at least 2 nodes have a copy of the data, read your data from any node and routing of your requests will happen magically (well, multicast, versionning of data, but let's call it "magic").
Want more performance ? add a node and reads will hit multiple nodes & use some map-reduce. Cassandra was benchmarked to out-perform mysql at 8 nodes. Yes, it means 8 1U low-cost servers to outperform a single mysql. But now you can scale easily, without requiring expensive hardware.
Want more availability ? add a node and set your "transaction safety level" to more nodes than 2.

LMTP over tcp daemon


This one would be the second daemon. It would handle lmtp, apply advanced filtering, interrogate cassandra for similar data to perform conversation grouping.
This one doesn't store anything and would be completly stateless. Deploy as much as needed to handle your postfix load. Just hide those behind some basic DNS round-robin.

IMAP connector


This one is needed to keep thunderbird in the loop. Reads cassandra data. IMAP client was a 20days job, so this one shouldn't be too hard. Just deploy some of them with DNS round-robin.
MiniG backend would hit cassandra nodes directly.

No code available on the net yet, but I want to try a messaging system built this way. This is just a personnal "will try" project. PostgreSQL, Cyrus+Murder & Heartbeat are not out of the loop... yet :D

Edit : to give some obm relevance to this blog entry, Adrien added email support to o-push. Works fine on Win Mobile, very slow on Nicolas's Android 2.x (htc magic with custom firmware for native exchange support), very fast on my iPhone, mixed results on Mehdi's Nokia E71. Expect a blog entry dedicated to that once first round of fixes hits googlecode.