I hadn’t really considered using EC2 since we had more servers in colocation than I really needed. But I had a file storage problem. When you have a thousand files, you stick them in a directory. When you have a million files, you cannot simply stick them in a single directory. You distribute them across multiple directories. What a PITA.

My first thought was to use MogileFS. It handles the directory hashing for you and distributes redundant copies of files across multiple servers. I had extra servers. Sweet. But before I rushed off and started building my shiny new filesystem, I wanted to check out the competitors. That led me to SmugMug. And that led me to S3.

I work at a tiny startup. I had a problem and very few developers to ask for help. Every hour I needed from was a significant impact on another project. And dammit, all the open projects were on fire. I needed to solve my file system problem and fast.

So up on S3 the files went. XML files. Beaucoup XML files.

It was painless. It was simple. It was cheap. The monthly S3 cost is a fraction of a server’s cost in colocation. Sweet!

Wait! If that’s so yummy, why not move XML processing up to EC2? Our XML processing load was increasing…increasingly increasing. I rewrote our XML processing app, built a custom amazon machine image (centos + apache + tomcat) and fired it up. Nice!

Building the machine instance was a pain but worth the effort. I learned a lot about centos that I didn’t previously know or really understand. However, I wish I had a real system administrator on staff. It would have hurt less.

One of the goals for the EC2-based XML processing was to shift from offline XML processing to a RESTful web service. That is, rather than queue the XML processing in a single process, I needed to finish the XML processing during the HTTP request. On demand processing. Done in seconds (not tens of minutes). And handle multiple concurrent processing requests.

Here is the EC2 <--> S3 connection. For each file received for processing, I write dozens to hundreds of files to S3 plus open scads of HTTP connections to other web servers. Running these in a single thread burned precious time. Even though we “write” to S3, the underlying mechanism is another HTTP request.

Simple. Build a thread pool for the HTTP requests and run multiple threads concurrently. That worked swimmingly but for one issue. It didn’t take long until I started seeing the “Too many open files” in the exception logs.

Normally, the limit on open files is quite adequate. But you bolt Apache’s HttpClient to the backend of your webapp and supercharge it with a healthy thread pool and you will overwhelm the default settings. Centos will not “garbage collect” the spent files from completed HTTP requests fast enough.

The solution: Up the limits on open files. The default is 1024. Simply edit /etc/security/limits.conf and change the soft and hard values for nofile. I’m sure there is a maximum size but these values have been working for me. What’s appropriate for your system is dependent on your system. You will need to pick size values for yourself.

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4
*                soft    nofile          8192
*                hard    nofile          65536

What was the net result of moving XML processing and storage up to the Amazon Cloud? Retired 60% of the servers in colocation. Built a scalable infractructure. Reduced overall monthly hosting costs. Fewer moving parts.

Now, if only I had a system administrator…

Early this spring, I made some big architectural changes in the company’s website. The two most far-reaching changes involved Amazon’s Web Services and Adobe’s Flex product. Sometimes you regret big changes. I only regret having not made the changes earlier.

I admit that I wasn’t always a flex fan. Indeed, I dismissed the flex out of hand in the early days mainly as a response to Macromedia’s steep pricing model. Ouch. Since then, Adobe had bought Macromedia and the pricing model changed several times. I never noticed. Such is the cost of writing something off.

Then I saw Peter Armstrong’s presentation at the Third Annual Silicon Valley Ruby Conference (April 18-19, 2008). Turns out, flex is now free. (uhhhmmm, ok. you will want to buy the developer ide but it’s possible to use the free command line compiler.)

I bought Armstrong’s book and started working through the tutorials. By the time I finished Chapter 6, I knew that I was going to change our webapp from a javascript-driven front-end to an flex-based, actionscript-driven front-end. We deal with a lot of tabulated data. Over a million sets and growing. Some of the datasets are small and some are quite large. Furthermore, each row had numerous event listeners attached. We had plans to add more, add mashups, add slicky bells and wistles. The larger tables were dozens of megabytes large. Which isn’t a problem in and of itself except that…

Javascript sucks.

Well, not really. It’s the browser’s that suck. All of them. We had to back off functionality just to get IE to render. Firefox’s javascript+dom engine (including ff3) would render the tables (if it didn’t crash) but not all the event listeners worked. Safari (web-kit) would render and all events worked but then Apple released the 5-second script warning. We schemed endlessly on how to break the problem up. Mostly we hoped one of the miracle javascript libraries would solve these problems. We knew it was unlikely but we hoped.

I made some side-by-side comparisons on my MacBookPro and actionscript beat javascript hands down. Orders of magnitude faster to load and render. Less code. Consistency between browsers. And finally, actionscript/flex worked. And worked well.

What didn’t work well was our MySQL database. Our million datasets occupied hundreds of millions of database rows in scads of tables and hundreds of gigabytes of disk space. A 3ware RAID10 disk subsystem with master/slave configuration was in complete IO overload all the time. I had Matt from mysql come in to help us unravel our mess. This guy was good. He identified, explained, documented and proposed a prioritized list of solutions.

But I started thinking, why the hell do we even store the datasets in MySQL in the first place? The only time we ever change a dataset is when we replace it. The relational database solution had always been a problem and was getting worse by the day. Thinking like a mathematician, I decided to solve a different problem. Why not just save the datasets in flatfiles and overwrite them when they change? Just store them in flatfiles. Like pictures.

Of course, every solution brings its own problems. And storing a million flatfiles has its problem: you cannot store a million files in a single directory (not on Centos+ext3 you cannot). I recalled reading about mogileFS some time back. It presented what looked like a flat directory in which you could store millions of files in a single directory, distributed the save over multiple machines with redundancy, and so on. Looking at that solution lead me to smugFS. And to Don MacAskill’s presentation on Scalability. Crap! I said. I don’t need to host the flatfile’s at our colocation center. I’ll put them on Amazon S3. And While I’m at it, I’ll build a RESTful API on Amazon EC2 to import the datasets.

And my team has done it. We have completed the move to EC2/S3 and Flex. I couldn’t be happier. Armstrong’s book, Flexible Rails, lit the spark for our migration to Flexible Web Services.

I will write about some of the hurdles we’ve overcome in future posts. Some of the things we’ve solved are (i) building an EC2 image from scratch, (ii) compressing flat files in a format which actionscripts’ ByteArray can uncompress, (iii) using the struts2 REST plugin, (iv) XML parsing using StAX.