Wednesday, August 05, 2009

Something's broken in JNDI ?

Consider the following unit test :


package ldapcrap;

import java.util.Hashtable;

import javax.naming.NamingException;
import javax.naming.directory.DirContext;
import javax.naming.directory.InitialDirContext;

import junit.framework.TestCase;

@SuppressWarnings("unchecked")
public class CrapTest extends TestCase {

private Hashtable env;

@Override
protected void setUp() throws Exception {
super.setUp();

env = new Hashtable();

env.put("java.naming.factory.initial",
"com.sun.jndi.ldap.LdapCtxFactory");
env.put("java.naming.provider.url", "ldap://10.0.0.5");
env.put(DirContext.SECURITY_AUTHENTICATION, "simple");
}

public void testJNDIisCrap() throws NamingException {
DirContext ctx = null;
try {
ctx = new InitialDirContext(env);
} catch (Throwable t) {
fail("test failed");
} finally {
ctx.close();
}
}

public void testInALoop() throws NamingException {
long time;
int COUNT;

COUNT = 1000;
time = System.currentTimeMillis();
for (int i = 0; i < COUNT; i++) {
testJNDIisCrap();
}
time = System.currentTimeMillis() - time;
System.out.println("did "+COUNT+" in "+(time/1000)+"second(s).");

COUNT = 10000;
time = System.currentTimeMillis();
for (int i = 0; i < COUNT; i++) {
testJNDIisCrap();
}
time = System.currentTimeMillis() - time;
System.out.println("did "+COUNT+" in "+(time/1000)+"second(s).");

COUNT = 100000;
time = System.currentTimeMillis();
for (int i = 0; i < COUNT; i++) {
testJNDIisCrap();
}
time = System.currentTimeMillis() - time;
System.out.println("did "+COUNT+" in "+(time/1000)+"second(s).");
}

@Override
protected void tearDown() throws Exception {
super.tearDown();
}
}


Here's what it shows when openldap is running on 32bit debian etch:
did 1000 in 2second(s).
did 10000 in 28second(s).
did 100000 in 441second(s).

Something sounds wrong with those results as the unit test is linear : open/close, open/close. As long as we do not get 2, 20, 200 or something approaching, either my test code is wrong, or some server configuration is wrong.

The results on 64bit lenny are:
did 1000 in 2second(s).
did 10000 in 19second(s).

No third result, it broke and the test failed with some useless null pointer. Let's "strace -f" the slapd process on lenny :
did 1000 in 13second(s).
did 10000 in 382second(s).

Ok, will take too long and seems stalled. Will write strace output to a file "strace -f -o slapd.log".

Starts good :
did 1000 in 6second(s).
did 10000 in 61second(s).

But then...
did 100000 in 1049second(s).

I expected between 500 & 620 seconds (I hoped the jvm JIT will do its job). I used my computer in the meantime, but not that much.

Still analysing the strace output, but if someone knows what is wrong in my test...

Thursday, July 30, 2009

Deliver MiniG on cd-rom ?

The problem

MiniG interface is built with GWT, that's common knowledge. GWT compiles java to javascript and generates permutations.

A permutation is a static javascript compilation unit for a given browser and a given locale. MiniG supports 2 locales (french & english). GWT 1.7 supports six browser flavours, so the compiler generates 12 javascript files (all the ABCD123.cache.html in the gwt build).

When a GWT application is loaded, a bootstrap file runs (the minig.nocache.js). This file runs checks on the user browser and determines which permutation should be used. The bootstrap javascript is only 6.2K and is similar for allmost all gwt applications. Once the correct permutation is determined, minig.nocache.js instructs the browser to load the abcd123.cache.html for your browser.

Most of the time, this file is already cached by your browser, but when a new version is deployed... the complete minig interface is downloaded.

With GWT compiler 1.7.0, the minig interface stands between 472K and 481K. Now you obviously begin to understand why this blog post is named "Deliver on cd-rom ?".

Ok, 500K is... half a second on a low cost end-user internet connection. As a french provider commercial says : "il a free, il a tout compris". But sometime, users do not even have an 8MB dsl connection. That's the case for the MiniG install running on my work mailbox. The 500K are a pain in the (insert joke about the G in MiniG) to download, as our work connection often uploads at 10K/s during work hours. This means that you cannot log into MiniG in less than 50 seconds.

Apart from getting correct network connectivity, I must have a look at what can be done on the MiniG side.

If you got me correctly, the localized messages are copied into javascript permutations. So if I add a 20K piece of text for online help in MiniG, the download size will go from 500K to 520K.

How can this be solved ?

Solution 1

Here is an easy one : a2enmod deflate

Run this command on the reverse proxy serving minig files and restart apache : gzip compression on static data files.

Before :

-rw-r--r-- 1 tom tom 479K 2009-07-30 00:41 DCEC18F66460AB2A6CB58F4EF8A75E70.cache.html


After :

-rw-r--r-- 1 tom tom 139K 2009-07-30 00:41 DCEC18F66460AB2A6CB58F4EF8A75E70.cache.html.gz


Just adding an apache module turned a 479K download to a 139K download. If the previous one was a 50sec download with our crappy bandwidth, this one is now 14sec. Pretty nice ?

Other solutions

MiniG is a pretty big application, but imagine something like Google Wave. You can guess they had to solve the same problem. They talked about it and provided solutions at the last Google I/O conference.

Let's summarize (or emphasize...) what the upcoming solutions are.

Compiler cut points

Even if minig never loads a page, all screens are not visible at the same time. There's no need to load the composer code when the frontpage only shows a list of message.

before :

addTab(Webmail.COMPOSER, new MailComposer());


after :

GWT.runAsync(new RunAsyncCallback() {
public void onSuccess() {
addTab(Webmail.COMPOSER, new MailComposer());
}
});


This tells the compiler that adding the composer to the UI is not "top priority". If this is the first usage of the MailComposer class (in MiniG case it is), all its code will be generated in a separate javascript file. The composer code will be loaded later, when the INBOX was already shown to the user.

Same applies for the progress bar shown on minig loading. We show the progress bar and can "runAsync" the XmlHttpRequest that fetches user settings on startup. Using this gives us :

  • Instant progress bar display

  • The main download is done with the progress bar on screen

  • The 500K become 50K and then 450K



With some runAsync added, I managed to display inbox with only a 170k download.

Class metadata

GWT obfuscates a lot of things (renames all your variables to a, b, c, z, a1, z26 instead of their original names). But it keeps the name of your java classes in the code. The -XdisableClassMetadata forces obfuscation of that to. MiniG uses fr.aliasource.webmail.*. This compile option gives more than 5% improvement. Easy win :-)

SOYC : Story Of Your Compile

This is a reporting tool designed to tell you which parts of your code contributes to the final JS size. This is really helpful to understand where runAsync and code refactoring are most needed.

Conclusion

All those improvements are available in GWT compiler trunk. I tried them all on MiniG code. It is pretty unstable right now, but the solutions are coming. Situation is not critical (500K for a complete application is nothing), but as MiniG will gain features, we had to investigate this. Other optimizations are coming in the compiler, Google I/O presentation is a very interesting read.

Thursday, July 16, 2009

Mobile device provisionning with o-push

I just commited mobile device provisionning to the o-push ActiveSync server. We tested it with an iPhone 3.0 and it allows pretty cool things.

What is mobile device provisionning :

When a device starts its dialog with the ActiveSync server, it sends an header "X-MS-PolicyKey : 0". The server can either ignore it or choose to enforce a security/enterprise policy. It (the server) does so by refusing the sync request and sending back a 449 http error code.

Of course you need a device that supports security policies, but the iPhone 3.0 does.

The device, when receiving the 449 error, starts a provisionning dialog :


  • it asks for the server policy

  • the server returns one with a temporary policy id

  • the device acknowledges the id by sending it back to server

  • the server respond with a "final policy id" for the device

  • the device will now use the "X-MS-PolicyKey: <final policy id>" in all its future communications

  • normal sync dialog will occur, except if the server administrator changes the policy (the servers sends another 449 error if the "final policy id" is no longer valid)



What can the administrator enforce in the policy :

Well, pretty much everything, but as the policy document is pretty explicit, lets reproduce it here :

<EASProvisionDoc>
<DevicePasswordEnabled>0</DevicePasswordEnabled>
<AlphanumericDevicePasswordRequired>0</AlphanumericDevicePasswordRequired>
<PasswordRecoveryEnabled>0</PasswordRecoveryEnabled>
<DeviceEncryptionEnabled>0</DeviceEncryptionEnabled>
<AttachmentsEnabled>1</AttachmentsEnabled>
<MinDevicePasswordLength>4</MinDevicePasswordLength>
<MaxInactivityTimeDeviceLock>900</MaxInactivityTimeDeviceLock>
<MaxDevicePasswordFailedAttempts>8</MaxDevicePasswordFailedAttempts>
<MaxAttachmentSize/>
<AllowSimpleDevicePassword>1</AllowSimpleDevicePassword>
<DevicePasswordExpiration/>
<DevicePasswordHistory>0</DevicePasswordHistory>
<AllowStorageCard>1</AllowStorageCard>
<AllowCamera>1</AllowCamera>
<RequireDeviceEncryption>0</RequireDeviceEncryption>
<AllowUnsignedApplications>1</AllowUnsignedApplications>
<AllowUnsignedInstallationPackages>1</AllowUnsignedInstallationPackages>
<MinDevicePasswordComplexCharacters>3</MinDevicePasswordComplexCharacters>
<AllowWiFi>1</AllowWiFi>
<AllowTextMessaging>1</AllowTextMessaging>
<AllowPOPIMAPEmail>1</AllowPOPIMAPEmail>
<AllowBluetooth>2</AllowBluetooth>
<AllowIrDA>1</AllowIrDA>
<RequireManualSyncWhenRoaming>0</RequireManualSyncWhenRoaming>
<AllowDesktopSync>1</AllowDesktopSync>
<MaxCalendarAgeFilter>0</MaxCalendarAgeFilter>
<AllowHTMLEmail>1</AllowHTMLEmail>
<MaxEmailAgeFilter>0</MaxEmailAgeFilter>
<MaxEmailBodyTruncationSize>-1</MaxEmailBodyTruncationSize>
<MaxEmailHTMLBodyTruncationSize>-1</MaxEmailHTMLBodyTruncationSize>
<RequireSignedSMIMEMessages>0</RequireSignedSMIMEMessages>
<RequireEncryptedSMIMEMessages>0</RequireEncryptedSMIMEMessages>
<RequireSignedSMIMEAlgorithm>0</RequireSignedSMIMEAlgorithm>
<RequireEncryptionSMIMEAlgorithm>0</RequireEncryptionSMIMEAlgorithm>
<AllowSMIMEEncryptionAlgorithmNegotiation>2</AllowSMIMEEncryptionAlgorithmNegotiation>
<AllowSMIMESoftCerts>1</AllowSMIMESoftCerts>
<AllowBrowser>1</AllowBrowser>
<AllowConsumerEmail>1</AllowConsumerEmail>
<AllowRemoteDesktop>1</AllowRemoteDesktop>
<AllowInternetSharing>1</AllowInternetSharing>
<UnapprovedInROMApplicationList/>
<ApprovedApplicationList/>
</EASProvisionDoc>

As you see, you can completly "enterprise lock" the phone :


  • You can lock wifi, infrared or bluetooth usage

  • No photos (we tested this one, the iPhone supports it)

  • No text messaging

  • No phone unlock without password (we tested this one too on the iPhone), with a strict password policy

  • No mail/groupware account except the enterprise/ActiveSync one

  • Only allow sending SMIME signed emails

  • Force storage of email/groupware data in an encrypted storage

  • etc



Right now, the policy sent to the phones is harcoded in the o-push server, but will probably develop an interface to manage those policies ;-)

Friday, July 03, 2009

We had it first ;-)

MiniG had drag & drop of conversations into folder before GMail. But my implementation clearly sucked (which was not a problem as I find drag&drop an inefficient way to move mail around).

Gmail did it better, so I mimiced their user interface for dnd.

No screenshot, but a small video ;-)

Wednesday, June 03, 2009

Vacation in MiniG

Now that filters and sieve scripts can be managed from MiniG, vacation is the next step.

Here's what the form looks like :

Monday, May 25, 2009

You asked for it...

And here they come : filters in MiniG :-)

Cyrus sieve client API is fully functionnal. The UI to define the filters is being implemented right now.

Mandatory screenshot below.

Saturday, April 25, 2009

o-push updates and minig status

Just did some updates on the o-push website to reflect its status. To sum it up, it's going pretty well.

The bad thing about server side software is you cannot show a screenshot. Will try to make a screencast of a working synchronisation.

Next o-push task is storage implementation : storing device ids, client and server mappings (obm database id to pda identifier). Right now this is stored in memory, but as we plan to integrate o-push in OBM 2.3, I'll use PostgreSQL tables for the persistent storage implementation.

On the MiniG front, Adrien just committed lots of work on iCalendar invitations. When you receive an event invitation, you can accept it and the event will be created in OBM (or updated if it was already coming from obm). In my opinion, one the nicest minig feature with event invitations is the "has:invitation" search tip to find your pending event invitations.