Bash on Red Leopard

filenames with spaces

Fri, 15 Oct 2010 08:48:22 +0000

I downloaded Unity last week. My first bit of installation geekery was to push the documentation to marmaduke, my CentOS server. I drop the documentation directory into a virtual host and let apache serve the index (since I solved my Apache Directory Indexing problem a bit back.)

However, I found that the pages had missing images. Looking at the HTML source, I found that some of the image URLs used ‘images’ (lower case i) while others used ‘Images’ (upper case I). To make matters worse, CSS and javascript files were also stored in the images directory.

bash uuid generator

Thu, 25 Mar 2010 04:20:50 +0000

Onliner bash scripts are handy but bash and common utilities don’t always work the same on the two systems I most use: Centos vs. OS X.

centos $ cat /etc/redhat-release 
CentOS release 5.4 (Final)

osx $ sw_vers | head -n2
ProductName:	Mac OS X
ProductVersion:	10.6.2

For example, I recently wrote a simple script to generate a set of UUID using the uuidgen utility. OS X and Centos versions of uuidgen take very different parameters.

bash date tricks

Sun, 16 Aug 2009 21:14:39 +0000

A quick usage note regarding the date util under bash. I sometimes want to convert between a unix timestamp and a formatted date string. I do it infrequently enough that I forget the syntax. This article is me writing down my notes.

In the following example, I want to get timestamps and date strings for both today and yesterday. Why yesterday’s date? Because I want to get yesterday’s data from google analytics’ data API. I’ve see numerous examples getting day, month and year then subtracting one from the day and propagating the underflow through the month and year. Blech! If I have today’s timestamp, I simply subtract a days worth of seconds from today and violà, yesterday!

use curl for api documentation

Thu, 02 Apr 2009 18:08:03 +0000

I’ve been working quite a bit with the rest plugin for Struts2. The really nice thing about this plugin is the way it cleans up Struts URLs. Makes them more rails-like. I chuckled when depressed programmer suggested that struts2 is “WebWork on drugs.” I hate struts2. I really do.

Anyway, I have stripped down an AccountController to show just the POST service. In reality, the create() method is wired to a middle tier service that authenticates username, password pairs then updates session attributes with member id and other bits of persistent session data I need.

bash progress monitor

Sat, 10 Jan 2009 05:55:28 +0000

I have a remote machine that is used to store and process XML files. Recently, I had need to duplicate a directory of XML files (e.g., cp -r a b). It’s not really germane to the subject here, but this particular server has a whack configuration and I gotta rant before I continue.

The office server (scrappy) has pretty good specs.

[scrappy ~]$ cat /proc/meminfo
MemTotal:      3980800 kB
[scrappy ~]$ cat /proc/cpuinfo

grep and UTF-8

Wed, 24 Dec 2008 01:44:24 +0000

I needed to look up the various strings Apple uses to name the iTunes Library. First I tried to get name from the iTunes resource bundle

echo "this won't work..." echo "so don't even try it"

cd /Applications/iTunes.app/Contents/Resources/English.lproj cat Localizable.strings | grep ‘PrimaryPlaylistName’

But I quickly learned that grep doesn’t work on the strings file. Why? Because Apple string files are not UTF-8. They are UTF-16. Usually. But in this case they are. I wanted to iterate over the set of resource strings and extract just the string I wanted.

centos l10n problem

Thu, 18 Dec 2008 02:46:03 +0000

Just about the time I believe the UTF-8 beast is in the cage, it escapes and runs amok.

This AM, I started to deploy an update to the webapp on EC2. Seems that some of the static strings in the app contained UTF-8 encoded non-ascii characters. The java compiler barfed. “The heck?”, I thought. I just compiled the app on my MacBook. I checked the usual suspects (tomcat’s server.xml, JAVA_OPTS) but everything looked fine. However, when I looked at the code, it was indeed mangled.

bash array crawler

Thu, 18 Dec 2008 02:18:06 +0000

I wanted to complement my bash directory crawler post with a bash array crawler example.

Sometimes, it’s easier to jack a list of identifying tokens into an array and process them rather than to build an end-to-end script with database access. For this contrived example, I grab a list of UUID from MySQL with a simple SQL statement.

mysql> SELECT id, uuid FROM icons;
+-----+--------------------------------------+
| id  | uuid                                 |
+-----+--------------------------------------+
|   1 | fe0b16ed-3369-4dda-8e60-faffb966375d |
|   3 | 82bfcbc2-84a2-4ca7-914b-13172b94feb6 |
|   6 | ab5e7265-3698-4205-b081-e6aec528fee2 |
|  11 | 4b6ca26b-c6ed-494f-aeb4-9bf369e2d465 |
|  19 | e7cc807b-7f15-46fa-b1c5-85d1f1050155 |
+-----+--------------------------------------+
5 rows in set (0.00 sec)

Next, jack the tokens into an array and simply crawl over the tokens.

EC2 and S3 Success Story

Thu, 11 Dec 2008 02:09:36 +0000

I’ve been building systems lately on Amazon’s Elastic Compute Cloud (EC2). At first, I was only interested in Amazon’s Simple Storage Solution (S3) after seeing the SmugMug slide show.

I hadn’t really considered using EC2 since we had more servers in colocation than I really needed. But I had a file storage problem. When you have a thousand files, you stick them in a directory. When you have a million files, you cannot simply stick them in a single directory. You distribute them across multiple directories. What a PITA.

bash directory crawler

Thu, 04 Dec 2008 19:44:21 +0000

Currently, popular filesystems (ext3, hfs+) have a practical limit on the number of files and directories you can store in a single directory. Certainly, most of the unix command line tools will not work once you exceed some magic threshold. In my experience, 10,000 files and or directories is the practical limit.

So what do you do when you have 1,000,000 XML files to process? I had this very problem recently. Fortunately, the problem was simplified as each file belong to one of 27,000 categories.

Fuser Detects FTP Completion

Tue, 25 Nov 2008 19:28:57 +0000

At work, we have legacy systems with problems which no one had taken the time to fix. One such legacy problem involved an FTP server. Client applications would FTP files up to the server for processing. That part worked fine. What didn’t work was knowing when the FTP was complete so we could start processing the data.

Recently, I decided to fix this problem.

Many people have written on the subject. One of the approaches advised, “watching the file and when the file size stops changing, you can use it.” I didn’t like that one. For so many reasons. Another recommended using lsof. Hummmmmmm. I didn’t get a warm fuzzy feeling with that one either.