In my investigation, I walked byte-by-byte through numerous binary dumps. Somewhere along the way I noticed a pattern. Forget about the head and foot bytes, the basic GZIP compressed data is simply not the same as the actionscript base deflate-algorithm compressed data.

People argue with me on that one. They say the compressed bytes are simply deflate-algorithm compressed and that these bytes can be decompressed by actionscript. I have but one comment. “Good luck with that, Cowboy. Knock yourself out.”

I have working code in production. Who you gonna believe?

I’ll start with the flex side first (flex3, as3). Use a URLLoader and set the dataFormat to BINARY or it will not work. Since the source data is not GZIP, I’ve made up a new file extension ‘xmlz’.

The code in listing 1 fetches the compressed XML, decompresses it and dumps it in a TextArea.

Listing 1. Simple Flex app to load compressed XML

<?xml version="1.0" encoding="utf-8"?>
<mx:Application
    xmlns:mx="http://www.adobe.com/2006/mxml"
    layout="vertical"
    backgroundGradientColors="[#ffffff, #c0c0c0]"
    width="100%"
    height="100%">

  <mx:Script>
  <![CDATA[
    import flash.utils.ByteArray;

    [Bindable]
    private var bytes:String = "";

    private var loader:URLLoader = new URLLoader();

    private function urlLoaderSend():void
    {
      loader.addEventListener(Event.COMPLETE, setResult);
      loader.dataFormat = URLLoaderDataFormat.BINARY;
      loader.load(new URLRequest("http://example.com/data.xmlz"));
    }

    private function setResult(event:Event):void
    {
      var ba:ByteArray = loader.data as ByteArray;
      ba.position = 0;
      ba.uncompress();
      bytes = ba.toString();
    }
  ]]>
  </mx:Script>

  <mx:Button
      label="get playlist"
      click="urlLoaderSend()" />
  <mx:TextArea
      id="resultTA"
      width="600"
      height="600"
      text="{bytes}" />
</mx:Application>

The server-side java is quite simple. I use the StAX XMLStreamWriter to generate XML. The writer’s constructor takes an OutputStream as an argument. Simply wrap the original OutputStream in a DeflaterOutputStream. The trick is to explicitly create the Deflater. If you don’t, the default will produce bytes incompatible with actionscript’s uncompress().

Listing 2. Java DeflaterOutputStream fragment (with StAX writer)

OutputStream out = new OutputStream();
    ⋮

Deflater d = new Deflater(Deflater.BEST_COMPRESSION, false);
OutputStream zipper = new DeflaterOutputStream(out, d);

XMLOutputFactory factory = XMLOutputFactory.newInstance();

XMLStreamWriter writer;
writer = factory.createXMLStreamWriter(zipper, "utf-8");
    ⋮

And that’s it! The compression is size-equivalent with GZIP (within a few bytes). I would rather actionscript supported GZIP. Until it does, I’ll continue using ‘xmlz’.

Early this spring, I made some big architectural changes in the company’s website. The two most far-reaching changes involved Amazon’s Web Services and Adobe’s Flex product. Sometimes you regret big changes. I only regret having not made the changes earlier.

I admit that I wasn’t always a flex fan. Indeed, I dismissed the flex out of hand in the early days mainly as a response to Macromedia’s steep pricing model. Ouch. Since then, Adobe had bought Macromedia and the pricing model changed several times. I never noticed. Such is the cost of writing something off.

Then I saw Peter Armstrong’s presentation at the Third Annual Silicon Valley Ruby Conference (April 18-19, 2008). Turns out, flex is now free. (uhhhmmm, ok. you will want to buy the developer ide but it’s possible to use the free command line compiler.)

I bought Armstrong’s book and started working through the tutorials. By the time I finished Chapter 6, I knew that I was going to change our webapp from a javascript-driven front-end to an flex-based, actionscript-driven front-end. We deal with a lot of tabulated data. Over a million sets and growing. Some of the datasets are small and some are quite large. Furthermore, each row had numerous event listeners attached. We had plans to add more, add mashups, add slicky bells and wistles. The larger tables were dozens of megabytes large. Which isn’t a problem in and of itself except that…

Javascript sucks.

Well, not really. It’s the browser’s that suck. All of them. We had to back off functionality just to get IE to render. Firefox’s javascript+dom engine (including ff3) would render the tables (if it didn’t crash) but not all the event listeners worked. Safari (web-kit) would render and all events worked but then Apple released the 5-second script warning. We schemed endlessly on how to break the problem up. Mostly we hoped one of the miracle javascript libraries would solve these problems. We knew it was unlikely but we hoped.

I made some side-by-side comparisons on my MacBookPro and actionscript beat javascript hands down. Orders of magnitude faster to load and render. Less code. Consistency between browsers. And finally, actionscript/flex worked. And worked well.

What didn’t work well was our MySQL database. Our million datasets occupied hundreds of millions of database rows in scads of tables and hundreds of gigabytes of disk space. A 3ware RAID10 disk subsystem with master/slave configuration was in complete IO overload all the time. I had Matt from mysql come in to help us unravel our mess. This guy was good. He identified, explained, documented and proposed a prioritized list of solutions.

But I started thinking, why the hell do we even store the datasets in MySQL in the first place? The only time we ever change a dataset is when we replace it. The relational database solution had always been a problem and was getting worse by the day. Thinking like a mathematician, I decided to solve a different problem. Why not just save the datasets in flatfiles and overwrite them when they change? Just store them in flatfiles. Like pictures.

Of course, every solution brings its own problems. And storing a million flatfiles has its problem: you cannot store a million files in a single directory (not on Centos+ext3 you cannot). I recalled reading about mogileFS some time back. It presented what looked like a flat directory in which you could store millions of files in a single directory, distributed the save over multiple machines with redundancy, and so on. Looking at that solution lead me to smugFS. And to Don MacAskill’s presentation on Scalability. Crap! I said. I don’t need to host the flatfile’s at our colocation center. I’ll put them on Amazon S3. And While I’m at it, I’ll build a RESTful API on Amazon EC2 to import the datasets.

And my team has done it. We have completed the move to EC2/S3 and Flex. I couldn’t be happier. Armstrong’s book, Flexible Rails, lit the spark for our migration to Flexible Web Services.

I will write about some of the hurdles we’ve overcome in future posts. Some of the things we’ve solved are (i) building an EC2 image from scratch, (ii) compressing flat files in a format which actionscripts’ ByteArray can uncompress, (iii) using the struts2 REST plugin, (iv) XML parsing using StAX.