A Week in the Slife
Monday’s morning scrum was dragging out. Scrum is supposed to be a short meeting so when it starts to drag, I’ll make a “now and later” request. What conversations can we complete “now” in a few minutes and what conversations are better served with their own “later” in their own planning meeting. I hadn’t scheduled Monday planning meetings. Sometimes meetings get in the way. Sometimes they are the way. Monday’s meetings were definitely overdue.
After lunch (where did I eat?), I completed final bug fixes and code cleanup for my development work on the Amazon EC2-based metadata processor. For each incoming XML file (from iTunes), we parse the metadata into multiple XML files (playlists), in multiple XML vocabularies (XSPF, Apple Plist) and store the files on Amazon’s S3. I added a CSV output writer so now we have both XML and CSV.
I updated the CollectionsController’s HTTP DELETE method. We use Struts2. I loathe Struts2 but it’s what we have. There is a REST plugin for Struts2 that makes the Action look and feel much like a Rails Controller.
So why not just use a RoR? I use the JAXP StAX parser. In my testing, StAX screams compared to REXML and I’m hooked on Java Collections. The XML documents we parse are large. StAX parses a 30MB XML document (including dereferencing all the playlist tracks) into a value object in under 2000ms.
I sat aside administrative time for the afternoon. Just under an hour.
Started a project to integrate Google Analytics event reporting into the metadata processor.
Google analytics rules. It is dead simple for web pages. Problem is, our Amazon EC2-based metadata processor doesn’t have web pages. It’s all server-side with a minimalist RESTful API. Web pages have javascript and most of google’s help is around adding javascript to your web page, calling google javascript functions from your web page. And so on.
I started with a couple articles (one, two) as my basis for building and threading server-side activity reporting to google sans javascript.
Mandarin classes started. I like the class. Class at 7:00 – 9:00 pm Tuesday night. makes for a long day. I also attend toastmasters 6:30 – 8:00 am Tuesday mornings.
Finished bulk of the coding for google analytics, threading the call in at just the right moment wasn’t hard. Constructing the query wasn’t hard. Understanding exactly what is needed in the query string took some reading.
I took time from coding in the afternoon for a code review of our web application. Aside from the Amazon EC2-based metadata processor, our web application is hosted at Silicon Valley Web Hosting. I am accountable for the development schedule on that web application as well. I needed to spend an afternoon with the developer working on that codebase.
At the end of the day, I pushed hard and finish my work on the analytics.
One of the concerns I have with our EC2-based metadata processor is running out of capacity. I created but one instance (albeit and extralarge instance) and it is handling our current load without a problem and can handle much more. At some point, I will want to add a second instance. And that means I’ll need a load balancer.
I almost never start a new project without seeing if someone has already solved it. I say, “almost” and when I don’t, I usually regret it. I’ve been looking at a number of load balancer articles. It came down to nginx and mod_proxy_balancer.
I chose mod_proxy_balancer. It seemed easier to deal with and–for our needs–more than adequate. The metadata processor doesn’t get a lot of requests. Basically, a client agent logs in to establish and authenticated session and then submits the metadata for processing. Two page hits and the session is done.
The authenticated session requirement necessitates a load balancer which handles sticky sessions. An article on scaling drupal on johnandcailin.com‘s blog is the basis for my load balancer. Since I’m using java and tomcat, I didn’t need to manipulate the cookies as the blog article describes. I simply needed to modify tomcat’s server.xml file to add a jvmRoute to the Engine element. I chose to add the hostname of the Amazon instance as the route.
I had an error in my load balancer configuration. I fat-fingered one of the characters in the route and that caused the balancer to “quietly” revert to round robin load balancing. I double checked all the routes, found the error and stickysession balance started working. I called it a day.
Before scrum, I cleaned files off my computer’s desktop.
After scrum, I finished documenting the load balancer setup on our wiki and shut down the load balancer. No need paying for what you don’t need.
After lunch, I logged into our servers at svwh for some long overdue maintenance.
Later, I started a new project which cross references members, artists and members. I first arrived at the algorithm last May but haven’t had the bandwidth to work on it. Probably a good thing. When I first thought of implementing the algorithm, I came at it from a C language background. Using binary keys and simple datastructures of pointers, the C approach would be extremely memory efficient and blazingly fast. When I started looking thinking about it on Friday, I had shifted to Java Collections approach. It will use more memory and run slower but I could implement it with very little code. Collections rule.
I have the basic algorithm written and will start working on data retrieval and storage next week.