Brown paper bag release

For the first time in minig history, x.y.z.1166 seems ok. Two new bug reports so far, one of them being that minig requires 1280x1024 for real use. The other one relates to "whole folder actions".

Biggest known bug in my list is "body part selection" : identify which part of an email is the body that must be shown to the user. Separating an email sent as "send as text and html" from a "text email with an html attachment".

All your folders are belong to us

My hate for a folder tree in MiniG never was a secret :

• It requires lot of screen real estate
• very hard to manage in a web context
• a tree helps for mail triage. When I began using internet, altavista was becoming a leader and yahoo was loosing marketshare. Google didn't exist. At this time yahoo was trying to "categorize" the web while altavista understood that trying to create a tree on top of internet-scale data was a lost fight.
• Displaying a tree is the easy solution : first thing I learned while attending Algorithm101 was that in computer science, everything can be represented with a tree.

A tree was clearly not the way I wanted to go. It's fine for a small bunch of data. It's also fine when you don't have any other way to do the triage of your data.

To make things short, folder tree is out of minig, for good. As I have to live with IMAP servers, I still had to find an efficient replacement. The replacement has the following features :

• dynamic displays that only shows folders with unread emails by default
• incremental search : a text field on top pushes the search paradigm of minig to your folders. Just type "m" in the field and you'll get your "MiniG" "Mini Me" "foo/mmm/bar" folders
• dynamic color assignment : each folders gets a color. This color is used when showing search results to easily identify where they are stored (in your inbox, in a shared folder, in your sent messages)

Even some of my users that felt in the "A folder tree on the left of a mail client is mandatory" category are saying some nice things about the new display.

BaraFolder (internal name of the new folder widget) (lot's of private jokes around that name) is available in the latest stable MiniG release we did today (2.2.15.1166). This release will hit about 1500 customer's mailboxes next week.

A release candidate of this minig version was shown to one of our customers last week and he said "no, I don't need the calendar demo, I want this" (ok, he was an easy target as we deployed horde/imp for him years ago). Given the time me, david and others dedicated into this release, I hope users will enjoy it.

Next steps are not set in stone, but big minig deployments (as in 8000 mailboxes) are planned. My work items should be :

• clustered deployement : N-frontend, M-backend
• bugfixes
• showing the filter feature in the UI. MiniG can write sieve filters, but it's disabled in released build as it conflicts with some obm features (vacation, forwarding, etc)
• bugfixes
• composer. this part of minig really needs some love. Even more when you see what google wave can do (no, the plan is not to be able to insert a "poll widget" in your signature)

PS : MiniG is free software, released as GPLv3. No paying-customers-only-crap-for-the-others version.

Random links of the weekend

Still exploring what Cassandra could provide as a clustered mail storage.

Easy read on cassandra data model : http://arin.me/code/wtf-is-a-supercolumn-cassandra-data-model

Using this article, I drafted an early data model :

Conversations : { // CF
    INBOX/123-1 : { // Row key is the conv id
 timeuuid_1 : messageId1
 timeuuid_2 : messageId2
    }      
    INBOX/123-2 : { // Row key is the conv id
         timeuuid_3 : messageId3
         timeuuid_4 : messageId4
    }       
}

Folders : { // CF
     INBOX : {
           timeuuid_1 : INBOX/123-1
           timeuuid_3 : INBOX/123-2
     }
}

Messages : { // CF
   messageId1 : { // row key
       subject: bla bla
     to: xx
     cc: yy
     bcc: zz
     preview : hello world
                   body: hello world ! qdqsd
                   hasAttach: true
                   hasInvitation: false
                   attachments: attach_123, attach_456
   }
}

Attachments : {
    attch_123 : {
        filename: fileName1
 size: 123456
        content: xazeqsdqdqsd_base_64
    }
}

Article on lucene/solr + cassandra : http://nicklothian.com/blog/2009/10/27/solr-cassandra-solandra/

As I wanted to store my tests on those technologies somewhere, I created the BigG project on googlecode.

My (not so) secret plan

Did some work on re-packing Cassandra as an OSGi based daemon. The secret plan is to have 3 eclipse products :

Cassandra node

Cassandra is the clustered DB. This kind of databases are designed with high availability in mind and operates in "multi-master" mode by default. Optimal configuration starts at 3 nodes : set your database to "validate transactions" when at least 2 nodes have a copy of the data, read your data from any node and routing of your requests will happen magically (well, multicast, versionning of data, but let's call it "magic").
Want more performance ? add a node and reads will hit multiple nodes & use some map-reduce. Cassandra was benchmarked to out-perform mysql at 8 nodes. Yes, it means 8 1U low-cost servers to outperform a single mysql. But now you can scale easily, without requiring expensive hardware.
Want more availability ? add a node and set your "transaction safety level" to more nodes than 2.

LMTP over tcp daemon

This one would be the second daemon. It would handle lmtp, apply advanced filtering, interrogate cassandra for similar data to perform conversation grouping.
This one doesn't store anything and would be completly stateless. Deploy as much as needed to handle your postfix load. Just hide those behind some basic DNS round-robin.

IMAP connector

This one is needed to keep thunderbird in the loop. Reads cassandra data. IMAP client was a 20days job, so this one shouldn't be too hard. Just deploy some of them with DNS round-robin.
MiniG backend would hit cassandra nodes directly.

No code available on the net yet, but I want to try a messaging system built this way. This is just a personnal "will try" project. PostgreSQL, Cyrus+Murder & Heartbeat are not out of the loop... yet :D

Edit : to give some obm relevance to this blog entry, Adrien added email support to o-push. Works fine on Win Mobile, very slow on Nicolas's Android 2.x (htc magic with custom firmware for native exchange support), very fast on my iPhone, mixed results on Mehdi's Nokia E71. Expect a blog entry dedicated to that once first round of fixes hits googlecode.

Innovation from mozilla team

Mozilla raindrop project just appeared on the web.

Like it or not, but at least Google with GMail & Wave is not the only one trying to innovate in email business any more.

Status update

OBM 2.3 freeze mode is on.

MiniG isn't branched, so I'm trying to add small features without destroying mail reading.

obm sync receives lots of performance related commits. We tweaked most of our sql queries to get good performance on both mysql & pg.

MySQL & PostgreSQL do not require the same kind of tuning. MySQL prefers 2 queries with select a from b ... and select c from b where b.id in (). PostgreSQL performs better with select c from b inner join a where... If we do the one query version postgresql wins & mysql crawls to death. If we do the 2 query version, mysql outperforms postgresql a little on low-cost hardware benchmarks.

We (re)did o-push tests on Nokia E71 with MailForExchange. The calendar part of o-push is working very well. We still need to take a look at contacts. We expect to release obm-2.3 rc0 with contacts & calendar working in o-push. Adrien is working on the mail part of o-push and discovered that microsoft has an alternative to locales. Forget iso-8859-1, let's call it 28591. utf-8 is 65001. o-push just won a new mapping table.

Still wondering if obm new contact screen will be ready for prime time. As it is, I don't like it. Knowing Mehdi & David skills with CSS & Javascript, I think we can still have a new killer module for OBM 2.3. Right now, removing MiniG contact screen to use the OBM one would get the "Over my dead body" response. But as we're all inspired by Snow Leopard Addressbook, Google Contacts and mobile me, we'll find a UI that satisfies everyone, including users ;-)

MiniG seems in a pretty good shape (lot's of people use it as their only mail software, including me). People on the obm mailing seems to have a hard time installing it. I need to figure out why :/

New round of MiniG changes

Hottest topic for minig was thread grouping. I did all the modifications to make it use Cyrus support for RFC5256. This is slower than the algorithm I came out with, but well my code was faster because it ignored the case that forces you to work with the "References" header.

"Slower" is something that should be taken with care. The load on minig backend JVM is greatly reduced. The "UID THREAD REFERENCES UTF-8 ALL" imap command forces cyrus to do the hard work. The new code replaces MiniG JVM load by Cyrus I/O load. As most sysadmins are OK to dedicate their SAN resources to cyrus, but tends to hate JaveVM, this ok : they will see a waiting JVM not sucking any resource and a cyrus crawling the SAN mounts to death :-)

I also fixed MiniG backend memory usage. MiniG works with conversations, IMAP servers with messages. MiniG needs to maintain a mapping between Conversation's identifiers and IMAP messages UIDs. This mapping was sucking lot of memory. MiniG already has an on-disk version of this mapping. Easy solution : Java SoftReference. A SoftReference is a Java reference that the JVM is allowed to collect when it is under memory pressure. Most of the time, minig will use the in-memory mapping. When memory pressure is to high, to cache will be evicted & rebuilt on demand. After 11years of Java programming, this was the first time I used SoftReference. I knew how strong/weak/soft/phantom references work in Java, but it's the first time my code cannot work without using them.

As it was "optimisation week-end", I also worked on MiniG bandwidth consumption. MiniG was already very good for remote access with limited bandwidth, but things could be made better.

When you display your inbox, every 20sec, minig javascript polls the backend for changes. This AJAX call downloaded the current page of messages & redrawn the grid. This transfer is in the 2KB range. I introduced versioning to minig caches. The AJAX call now sends its last know version to the frontend. Frontend relays it the backend. The backend answers with http 304 when no change occurs. Frontend sends a "UseCachedData" RuntimeException the javascript. The most common case "nothing changed" goes from a 2KB transfer to a 81byte transfer. Quite nice improvement.

For now this is only done for IMAP folders, not for search results. But I as bought the "SOLR 1.4 Enterprise Search Server" book (very good investment for solr users), their number one recommandation for solr performance is to enable "proxy support" which will make solr respond with http 304 when your last search gives the same result. That's why I started implementing support of HTTP 304 in minig backend & frontend. Adding 304 support to minig search will give a big performance boost to "Unread" mail fans.

Upcoming release also adds a user visible feature : when you read a conversation, all unread emails are auto-expanded.

Next bug on my list is about composer, iframes in design mode & cut'n'paste. After 2 years of minig work, I tend to hate those "browser dependent" bugs.

I'm still hoping to do a stable minig release this week with obm 2.2.14, the composer bug being the last known release blocker.

Status of my OBM parts before oct 15th freeze

Worked 4/5 days on making sure that obm-sync was performing correctly on a 13GB OBM 2.3 PostgreSQL database. I must thank again yourkit.com for the license they granted me for MiniG. Their software is not free (as in speech), but they give away licences for free softwares. This kind of ubber cool software just tells you when your code is shit.

Small commit on obm-caldav to prevent it from fetching all calendar permissions from database. Yourkit helped on this one too.

MiniG thread grouping still needed some work. Slashdot article from Cyrus most active contributors (fastmail.fm) gave me some hints. Answer "UID THREAD REFERENCES UTF-8 ALL". This simple IMAP command forces cyrus to do all the threading calculation for you. The minig part is still complicated but from a performance pov, it's a win.

Ok, I really depend on cyrus features. For today deadlines, that's fine. But the architecture I want to have for minig is :

incoming mail -> ironport (or any other _efficient_ spam filtering solution) -> minig_lmtp -> cassandra clustered db

Solr would be plugged into cassandra (cassandra is the big table implementation that facebook released to the apache group as free software).

Yes cyrus does not exist in the architecture I'm hoping to get minig to. (s|l)mtp connector to clustered db with full text indexer is google architecture. This architecture seems like the right one to me. murder + heartbeat seems pretty fragile when you compare it to cassandra/big table reliability.

Release often, Release early

Very early in some cases... We had to prepare a demo of o-push with obm 2.3 trunk (bleeding egde) and minig.

I had to stop working on minig temporarily to get o-push in a demo-able state.

OPush work was stalled by higher priority tasks, but after working on it all the week-end, it's in a pretty cool shape. I've just committed true push support for calendars.

Here is a demo video I did with my test setup : every modification in obm 2.3 calendar appears in the pda few seconds later.

(Yes, it's not perfect, I still don't know why I add to go back to the activesync application while recording)



PS: I had to convert the video to xvid as youtube seems to do shit with ogv's from gtk-recordMyDesktop

OBM Barcamp

We spent friday afternoon & saturday on a barcamp dedicated to OBM with some co-workers from Paris.

At the end of the first afternoon, I was pretty dissapointed :

• Lots of talking
• No actions

My first impression was that I lost an afternoon of minig bugfixing. The second day changed my opinion.

Mehdi & Guillaume merged most of the patches we maintained for our main production setup.

Sylvain worked with Erwan to figure out the best way to make LemonLDAP + OBM + MiniG work out of the box.

David & Vincent did some improvements on 2.3 calendar.

Tony worked on obm-automation improvements to allow running update.pl at login time for auto-provisionning of mailboxes & ldap entries.

I worked with Michel & Guillaume to figure out how to handle really big mailboxes in MiniG. Michel mailbox is a 20GB one. To give you an idea, our cyrus setup in Toulouse is only 35GB for 44 mailboxes & lots of mailshares.

Michel's mailbox was a very interesting test-case. It triggered a "too many open files" pretty rapidly. We identified a very old bug : the class that read minig configuration files left a file descriptor open.

This jumbo mailbox has other properties : it is big enough to trigger bugs that only occured on Toulouse production setup after more than 24h. As we are still unsure if the deadlock timeout that occurs on some jdbc transactions is derby's fault or minig's fault, I just committed the switch from a derby internal database to an external PostgreSQL instance. This change will permit one thing: identify if the bug was in derby.

Some easier to fix bugs were also identified :

• Advanced search misshandled searches on subject with two keywords
• Tony helped me fix the filter that was used for autocomplete when ldapContacts.pl is active

Some decision were taken:

• We confirmed that OBM feature & ui freeze will be effective on Oct 15th.
• Contact search in OBM & MiniG will move from SQL queries to a SOLR core (we're not sure yet if it will meet the 2.3 freeze date). We will also try to do the same for events.
• We are going to integrate a real LemonLDAP sso provider in minig. One that will not require OBM sso subsystem

If I consider the complete barcamp session, It was needed & productive.

Spinner of death (tm) Improvement

Sometime things can go wrong in MiniG, even with legitimate reasons :

• You suspended your laptop and your minig session is expired when your browser comes back to life


• Your beloved administrator restarted a tomcat server without notifying anybody


• Your network was down more than 3 minutes (MiniG frontend session duration)

This case will be handled in a next releases.



Yes, the texts are temporary and TartifletteMode is only how an expired session is called in the code.

(Temporary) Epic Fail

Changing MiniG mail grouping is more tricky than expected.

The previous algorithm worked like this :
- read the subject
- compute its root, "Fwd: Re: Hello sent the 2009-08-07" becomes "hello sent the xxxx-xx-xx"
- simply use an equality comparison on the subject roots.

First version with real threads used an hacked version of subject grouping. It was just a hack on the "equality comparison". Instead of comparing suject roots, it used lists of Message-ID headers and was doing comparisons on the differences between those 2. It worked. For mailboxes < 1000 messages. Sylvain's 100k mails Junk folder took more than 1hour to process. We needed a linear algorithm to do the thread grouping.

The new algorithm works on other headers : Message-ID and In-Reply-To.

It works this way. We maintain a ThreadRoot list which holds known conversations with the message-id's in them

When we detect that messages are added/changed or deleted to a folder, we first process removals. When a ThreadRoot has no more messages, we mark it as dead and flag it for removal.

Then we process updates, this is the tricky part.

We have 2 lists :
- List<ThreadRoot>, all the known thread roots not flagged as dead.
- a List<RawMessage> called leafCandidates where we have all the not yet processed messages.

We then run :

unmerged = 0;
merged = -1;
while (merged < unmerged) {
    unmerged = leafCandidates.size();
    doMerge();
    merged = leafCandidates.size();
}

Then doMerge() does all the job :

for (RawMessage r : leafCandidates) {
    if rootsIds.contains(r.messageId) {
       flagUpdate(threadRoot, r);
    } else if (r.inReplyTo == null) {
       createThreadRoot(r);
    } else {
       ThreadRoot tr = rootIds.get(r.inReplyTo);
       if (tr) {
         tr.merge(r);
       } else {
         // corner cases, most problems are not here
         // exemple : a mail with an In-Reply-To and the father mail is deleted
       }
    }
}

This algorithm is mostly linear... Its real complexity depends X, Y, Z where X is how much new mail you receive, Y how much you receive replies to existing emails and Z, how much the user changes flags on existing emails. It can easily process Sylvain 100k spams in 20sec, so let's assume the complixity is OK.

Most MiniG problems on our trunk version are in the "flagUpdate(threadRoot, r)" process. This creates all kind of strange bugs : conversations that still look read, mark as read/unread that only work "sometime".

Flags are still broken, but the new algorith is pretty promising. The load on our test & production mail server is lighter, really lighter. Some tuning of the last corner cases will make it a very good change. In fact the new code while still buggy is so fast that it exposed race conditions in the indexing code ;-)

Long time no blog

Massive changes are coming to minig. Real thread sorting, by message actions, etc.

We added a second minig installation to our test setup. We now have a MiniG setup running on our production mail server, and another one running on a lenny 64bit kvm. Both are using our production mail server.

The one running in kvm allows a first round of testing on live mailboxes. Performance testing is easier on this one, as the IMAP load and the java load are cleanly splitted on two servers.

With this new setup I already identified an easy optimisation : the select imap command forces Cyrus to write a log line. Storing the selected imap folder in the org.minig.imap lib might provide sensible speed improvements to initial mailbox indexing.

Big minig bug hunt

I'm focused on chasing minig bugs.

We clearly improved our testing flow. Releases are tested everyday by ff3 users, ff3.5 users and chromium users (Sylvain & me dropped firefox now that 64bit builds are available). IE8 is often tested on our TSE server. I still need to find an easy way to do IE6 testing.


What's coming into minig :

- real thread grouping. User testing shown that "subject+ignore numbers" is only manageable to technical people.

- composer improvements (I'm fighting with GWT RichTextArea, and learning things I didn't want to know about browsers differences).

- by email actions. Delete one, Star one, Print one. Add sender to contacts, etc.

- next/previous buttons. I don't read my email like that, but a lots of people seems to think that when you read a conversation, next has a meaning.

- folder management improvents. Rename & move.

And lots of needed bugfixes......

Something's broken in JNDI ?

Consider the following unit test :

package ldapcrap;

import java.util.Hashtable;

import javax.naming.NamingException;
import javax.naming.directory.DirContext;
import javax.naming.directory.InitialDirContext;

import junit.framework.TestCase;

@SuppressWarnings("unchecked")
public class CrapTest extends TestCase {

 private Hashtable env;

 @Override
 protected void setUp() throws Exception {
  super.setUp();

  env = new Hashtable();

  env.put("java.naming.factory.initial",
    "com.sun.jndi.ldap.LdapCtxFactory");
  env.put("java.naming.provider.url", "ldap://10.0.0.5");
  env.put(DirContext.SECURITY_AUTHENTICATION, "simple");
 }

 public void testJNDIisCrap() throws NamingException {
  DirContext ctx = null;
  try {
   ctx = new InitialDirContext(env);
  } catch (Throwable t) {
   fail("test failed");
  } finally {
   ctx.close();
  }
 }

 public void testInALoop() throws NamingException {
  long time;
  int COUNT;
   
  COUNT = 1000;
  time = System.currentTimeMillis();
  for (int i = 0; i < COUNT; i++) {
   testJNDIisCrap();
  }
  time = System.currentTimeMillis() - time;
  System.out.println("did "+COUNT+" in "+(time/1000)+"second(s).");

  COUNT = 10000;
  time = System.currentTimeMillis();
  for (int i = 0; i < COUNT; i++) {
   testJNDIisCrap();
  }
  time = System.currentTimeMillis() - time;
  System.out.println("did "+COUNT+" in "+(time/1000)+"second(s).");

  COUNT = 100000;
  time = System.currentTimeMillis();
  for (int i = 0; i < COUNT; i++) {
   testJNDIisCrap();
  }
  time = System.currentTimeMillis() - time;
  System.out.println("did "+COUNT+" in "+(time/1000)+"second(s).");
}

 @Override
 protected void tearDown() throws Exception {
  super.tearDown();
 }
}

Here's what it shows when openldap is running on 32bit debian etch:
did 1000 in 2second(s).
did 10000 in 28second(s).
did 100000 in 441second(s).

Something sounds wrong with those results as the unit test is linear : open/close, open/close. As long as we do not get 2, 20, 200 or something approaching, either my test code is wrong, or some server configuration is wrong.

The results on 64bit lenny are:
did 1000 in 2second(s).
did 10000 in 19second(s).

No third result, it broke and the test failed with some useless null pointer. Let's "strace -f" the slapd process on lenny :
did 1000 in 13second(s).
did 10000 in 382second(s).

Ok, will take too long and seems stalled. Will write strace output to a file "strace -f -o slapd.log".

Starts good :
did 1000 in 6second(s).
did 10000 in 61second(s).

But then...
did 100000 in 1049second(s).

I expected between 500 & 620 seconds (I hoped the jvm JIT will do its job). I used my computer in the meantime, but not that much.

Still analysing the strace output, but if someone knows what is wrong in my test...

Deliver MiniG on cd-rom ?

The problem

MiniG interface is built with GWT, that's common knowledge. GWT compiles java to javascript and generates permutations.

A permutation is a static javascript compilation unit for a given browser and a given locale. MiniG supports 2 locales (french & english). GWT 1.7 supports six browser flavours, so the compiler generates 12 javascript files (all the ABCD123.cache.html in the gwt build).

When a GWT application is loaded, a bootstrap file runs (the minig.nocache.js). This file runs checks on the user browser and determines which permutation should be used. The bootstrap javascript is only 6.2K and is similar for allmost all gwt applications. Once the correct permutation is determined, minig.nocache.js instructs the browser to load the abcd123.cache.html for your browser.

Most of the time, this file is already cached by your browser, but when a new version is deployed... the complete minig interface is downloaded.

With GWT compiler 1.7.0, the minig interface stands between 472K and 481K. Now you obviously begin to understand why this blog post is named "Deliver on cd-rom ?".

Ok, 500K is... half a second on a low cost end-user internet connection. As a french provider commercial says : "il a free, il a tout compris". But sometime, users do not even have an 8MB dsl connection. That's the case for the MiniG install running on my work mailbox. The 500K are a pain in the (insert joke about the G in MiniG) to download, as our work connection often uploads at 10K/s during work hours. This means that you cannot log into MiniG in less than 50 seconds.

Apart from getting correct network connectivity, I must have a look at what can be done on the MiniG side.

If you got me correctly, the localized messages are copied into javascript permutations. So if I add a 20K piece of text for online help in MiniG, the download size will go from 500K to 520K.

How can this be solved ?

Solution 1

Here is an easy one : a2enmod deflate

Run this command on the reverse proxy serving minig files and restart apache : gzip compression on static data files.

Before :

-rw-r--r-- 1 tom tom 479K 2009-07-30 00:41 DCEC18F66460AB2A6CB58F4EF8A75E70.cache.html

After :

-rw-r--r-- 1 tom tom 139K 2009-07-30 00:41 DCEC18F66460AB2A6CB58F4EF8A75E70.cache.html.gz

Just adding an apache module turned a 479K download to a 139K download. If the previous one was a 50sec download with our crappy bandwidth, this one is now 14sec. Pretty nice ?

Other solutions

MiniG is a pretty big application, but imagine something like Google Wave. You can guess they had to solve the same problem. They talked about it and provided solutions at the last Google I/O conference.

Let's summarize (or emphasize...) what the upcoming solutions are.

Compiler cut points

Even if minig never loads a page, all screens are not visible at the same time. There's no need to load the composer code when the frontpage only shows a list of message.

before :

    addTab(Webmail.COMPOSER, new MailComposer());

after :

    GWT.runAsync(new RunAsyncCallback() {
        public void onSuccess() {
           addTab(Webmail.COMPOSER, new MailComposer());
        }
    });

This tells the compiler that adding the composer to the UI is not "top priority". If this is the first usage of the MailComposer class (in MiniG case it is), all its code will be generated in a separate javascript file. The composer code will be loaded later, when the INBOX was already shown to the user.

Same applies for the progress bar shown on minig loading. We show the progress bar and can "runAsync" the XmlHttpRequest that fetches user settings on startup. Using this gives us :

• Instant progress bar display


• The main download is done with the progress bar on screen


• The 500K become 50K and then 450K

With some runAsync added, I managed to display inbox with only a 170k download.

Class metadata

GWT obfuscates a lot of things (renames all your variables to a, b, c, z, a1, z26 instead of their original names). But it keeps the name of your java classes in the code. The -XdisableClassMetadata forces obfuscation of that to. MiniG uses fr.aliasource.webmail.*. This compile option gives more than 5% improvement. Easy win :-)

SOYC : Story Of Your Compile

This is a reporting tool designed to tell you which parts of your code contributes to the final JS size. This is really helpful to understand where runAsync and code refactoring are most needed.

Conclusion

All those improvements are available in GWT compiler trunk. I tried them all on MiniG code. It is pretty unstable right now, but the solutions are coming. Situation is not critical (500K for a complete application is nothing), but as MiniG will gain features, we had to investigate this. Other optimizations are coming in the compiler, Google I/O presentation is a very interesting read.

Mobile device provisionning with o-push

I just commited mobile device provisionning to the o-push ActiveSync server. We tested it with an iPhone 3.0 and it allows pretty cool things.

What is mobile device provisionning :

When a device starts its dialog with the ActiveSync server, it sends an header "X-MS-PolicyKey : 0". The server can either ignore it or choose to enforce a security/enterprise policy. It (the server) does so by refusing the sync request and sending back a 449 http error code.

Of course you need a device that supports security policies, but the iPhone 3.0 does.

The device, when receiving the 449 error, starts a provisionning dialog :

• it asks for the server policy


• the server returns one with a temporary policy id


• the device acknowledges the id by sending it back to server


• the server respond with a "final policy id" for the device


• the device will now use the "X-MS-PolicyKey: <final policy id>" in all its future communications


• normal sync dialog will occur, except if the server administrator changes the policy (the servers sends another 449 error if the "final policy id" is no longer valid)

What can the administrator enforce in the policy :

Well, pretty much everything, but as the policy document is pretty explicit, lets reproduce it here :

<EASProvisionDoc>
<DevicePasswordEnabled>0</DevicePasswordEnabled>
<AlphanumericDevicePasswordRequired>0</AlphanumericDevicePasswordRequired>
<PasswordRecoveryEnabled>0</PasswordRecoveryEnabled>
<DeviceEncryptionEnabled>0</DeviceEncryptionEnabled>
<AttachmentsEnabled>1</AttachmentsEnabled>
<MinDevicePasswordLength>4</MinDevicePasswordLength>
<MaxInactivityTimeDeviceLock>900</MaxInactivityTimeDeviceLock>
<MaxDevicePasswordFailedAttempts>8</MaxDevicePasswordFailedAttempts>
<MaxAttachmentSize/>
<AllowSimpleDevicePassword>1</AllowSimpleDevicePassword>
<DevicePasswordExpiration/>
<DevicePasswordHistory>0</DevicePasswordHistory>
<AllowStorageCard>1</AllowStorageCard>
<AllowCamera>1</AllowCamera>
<RequireDeviceEncryption>0</RequireDeviceEncryption>
<AllowUnsignedApplications>1</AllowUnsignedApplications>
<AllowUnsignedInstallationPackages>1</AllowUnsignedInstallationPackages>
<MinDevicePasswordComplexCharacters>3</MinDevicePasswordComplexCharacters>
<AllowWiFi>1</AllowWiFi>
<AllowTextMessaging>1</AllowTextMessaging>
<AllowPOPIMAPEmail>1</AllowPOPIMAPEmail>
<AllowBluetooth>2</AllowBluetooth>
<AllowIrDA>1</AllowIrDA>
<RequireManualSyncWhenRoaming>0</RequireManualSyncWhenRoaming>
<AllowDesktopSync>1</AllowDesktopSync>
<MaxCalendarAgeFilter>0</MaxCalendarAgeFilter>
<AllowHTMLEmail>1</AllowHTMLEmail>
<MaxEmailAgeFilter>0</MaxEmailAgeFilter>
<MaxEmailBodyTruncationSize>-1</MaxEmailBodyTruncationSize>
<MaxEmailHTMLBodyTruncationSize>-1</MaxEmailHTMLBodyTruncationSize>
<RequireSignedSMIMEMessages>0</RequireSignedSMIMEMessages>
<RequireEncryptedSMIMEMessages>0</RequireEncryptedSMIMEMessages>
<RequireSignedSMIMEAlgorithm>0</RequireSignedSMIMEAlgorithm>
<RequireEncryptionSMIMEAlgorithm>0</RequireEncryptionSMIMEAlgorithm>
<AllowSMIMEEncryptionAlgorithmNegotiation>2</AllowSMIMEEncryptionAlgorithmNegotiation>
<AllowSMIMESoftCerts>1</AllowSMIMESoftCerts>
<AllowBrowser>1</AllowBrowser>
<AllowConsumerEmail>1</AllowConsumerEmail>
<AllowRemoteDesktop>1</AllowRemoteDesktop>
<AllowInternetSharing>1</AllowInternetSharing>
<UnapprovedInROMApplicationList/>
<ApprovedApplicationList/>
</EASProvisionDoc>

As you see, you can completly "enterprise lock" the phone :

• You can lock wifi, infrared or bluetooth usage


• No photos (we tested this one, the iPhone supports it)


• No text messaging


• No phone unlock without password (we tested this one too on the iPhone), with a strict password policy


• No mail/groupware account except the enterprise/ActiveSync one


• Only allow sending SMIME signed emails


• Force storage of email/groupware data in an encrypted storage


• etc

Right now, the policy sent to the phones is harcoded in the o-push server, but will probably develop an interface to manage those policies ;-)

We had it first ;-)

MiniG had drag & drop of conversations into folder before GMail. But my implementation clearly sucked (which was not a problem as I find drag&drop an inefficient way to move mail around).

Gmail did it better, so I mimiced their user interface for dnd.

No screenshot, but a small video ;-)

Vacation in MiniG

Now that filters and sieve scripts can be managed from MiniG, vacation is the next step.

Here's what the form looks like :

You asked for it...

And here they come : filters in MiniG :-)

Cyrus sieve client API is fully functionnal. The UI to define the filters is being implemented right now.

Mandatory screenshot below.

o-push updates and minig status

Just did some updates on the o-push website to reflect its status. To sum it up, it's going pretty well.

The bad thing about server side software is you cannot show a screenshot. Will try to make a screencast of a working synchronisation.

Next o-push task is storage implementation : storing device ids, client and server mappings (obm database id to pda identifier). Right now this is stored in memory, but as we plan to integrate o-push in OBM 2.3, I'll use PostgreSQL tables for the persistent storage implementation.

On the MiniG front, Adrien just committed lots of work on iCalendar invitations. When you receive an event invitation, you can accept it and the event will be created in OBM (or updated if it was already coming from obm). In my opinion, one the nicest minig feature with event invitations is the "has:invitation" search tip to find your pending event invitations.

O-Push, ActiveSync protocol implementation

MiniG is doing fine, so I am working on something else : an ActiveSync protocol implementation to replace OBM PDAs synchronisation layer.

The O-Push project is hosted on google code : http://code.google.com/p/o-push/.

Some things allready works : read-only calendar sync from OBM to the iPhone. O-Push can also send mail folder and test emails to the mobile device.

All your calendar are belong to us

Screenshot day

This time with the separate grid, always correctly sized, for dates labels & all day events ;-)

Better mockup with calendar view

It is not visible on the screenshot, but everything resizes correctly : not outer scrollbar, vertical space is occupied, the columns scale correctly.

Calendar mockup

As I'll probably never be OK with MiniG interface fitted into OBM, I'm experimenting on calendar integration into MiniG.

I'm just gathering infos for now, taking screenshots of Google Calendar, OSX iCal and Outlook Web Access. But as you can see on the following screenshot, MiniG looks nice with its iCal like portlet :D



You can also see on the screenshot that french i18n is available in current svn ;-)