New round of MiniG changes

Hottest topic for minig was thread grouping. I did all the modifications to make it use Cyrus support for RFC5256. This is slower than the algorithm I came out with, but well my code was faster because it ignored the case that forces you to work with the "References" header.

"Slower" is something that should be taken with care. The load on minig backend JVM is greatly reduced. The "UID THREAD REFERENCES UTF-8 ALL" imap command forces cyrus to do the hard work. The new code replaces MiniG JVM load by Cyrus I/O load. As most sysadmins are OK to dedicate their SAN resources to cyrus, but tends to hate JaveVM, this ok : they will see a waiting JVM not sucking any resource and a cyrus crawling the SAN mounts to death :-)

I also fixed MiniG backend memory usage. MiniG works with conversations, IMAP servers with messages. MiniG needs to maintain a mapping between Conversation's identifiers and IMAP messages UIDs. This mapping was sucking lot of memory. MiniG already has an on-disk version of this mapping. Easy solution : Java SoftReference. A SoftReference is a Java reference that the JVM is allowed to collect when it is under memory pressure. Most of the time, minig will use the in-memory mapping. When memory pressure is to high, to cache will be evicted & rebuilt on demand. After 11years of Java programming, this was the first time I used SoftReference. I knew how strong/weak/soft/phantom references work in Java, but it's the first time my code cannot work without using them.

As it was "optimisation week-end", I also worked on MiniG bandwidth consumption. MiniG was already very good for remote access with limited bandwidth, but things could be made better.

When you display your inbox, every 20sec, minig javascript polls the backend for changes. This AJAX call downloaded the current page of messages & redrawn the grid. This transfer is in the 2KB range. I introduced versioning to minig caches. The AJAX call now sends its last know version to the frontend. Frontend relays it the backend. The backend answers with http 304 when no change occurs. Frontend sends a "UseCachedData" RuntimeException the javascript. The most common case "nothing changed" goes from a 2KB transfer to a 81byte transfer. Quite nice improvement.

For now this is only done for IMAP folders, not for search results. But I as bought the "SOLR 1.4 Enterprise Search Server" book (very good investment for solr users), their number one recommandation for solr performance is to enable "proxy support" which will make solr respond with http 304 when your last search gives the same result. That's why I started implementing support of HTTP 304 in minig backend & frontend. Adding 304 support to minig search will give a big performance boost to "Unread" mail fans.

Upcoming release also adds a user visible feature : when you read a conversation, all unread emails are auto-expanded.

Next bug on my list is about composer, iframes in design mode & cut'n'paste. After 2 years of minig work, I tend to hate those "browser dependent" bugs.

I'm still hoping to do a stable minig release this week with obm 2.2.14, the composer bug being the last known release blocker.