Bash

I downloaded Unity last week. My first bit of installation geekery was to push the documentation to marmaduke, my CentOS server. I drop the documentation directory into a virtual host and let apache serve the index (since I solved my Apache Directory Indexing problem a bit back.) However, I found that the pages had missing images. Looking at the HTML source, I found that some of the image URLs used ‘images’ (lower case i) while others used ‘Images’ (upper case I). To make matters worse, CSS and javascript files were also stored in the images directory. ...

Onliner bash scripts are handy but bash and common utilities don’t always work the same on the two systems I most use: Centos vs. OS X. centos $ cat /etc/redhat-release CentOS release 5.4 (Final) osx $ sw_vers | head -n2 ProductName: Mac OS X ProductVersion: 10.6.2 For example, I recently wrote a simple script to generate a set of UUID using the uuidgen utility. OS X and Centos versions of uuidgen take very different parameters. ...

A quick usage note regarding the date util under bash. I sometimes want to convert between a unix timestamp and a formatted date string. I do it infrequently enough that I forget the syntax. This article is me writing down my notes. In the following example, I want to get timestamps and date strings for both today and yesterday. Why yesterday’s date? Because I want to get yesterday’s data from google analytics’ data API. I’ve see numerous examples getting day, month and year then subtracting one from the day and propagating the underflow through the month and year. Blech! If I have today’s timestamp, I simply subtract a days worth of seconds from today and violà, yesterday! ...

I’ve been working quite a bit with the rest plugin for Struts2. The really nice thing about this plugin is the way it cleans up Struts URLs. Makes them more rails-like. I chuckled when depressed programmer suggested that struts2 is “WebWork on drugs.” I hate struts2. I really do. Anyway, I have stripped down an AccountController to show just the POST service. In reality, the create() method is wired to a middle tier service that authenticates username, password pairs then updates session attributes with member id and other bits of persistent session data I need. ...

I have a remote machine that is used to store and process XML files. Recently, I had need to duplicate a directory of XML files (e.g., cp -r a b). It’s not really germane to the subject here, but this particular server has a whack configuration and I gotta rant before I continue. The office server (scrappy) has pretty good specs. [scrappy ~]$ cat /proc/meminfo MemTotal: 3980800 kB [scrappy ~]$ cat /proc/cpuinfo ...

I needed to look up the various strings Apple uses to name the iTunes Library. First I tried to get name from the iTunes resource bundle echo "this won't work..." echo "so don't even try it" cd /Applications/iTunes.app/Contents/Resources/English.lproj cat Localizable.strings | grep ‘PrimaryPlaylistName’ But I quickly learned that grep doesn’t work on the strings file. Why? Because Apple string files are not UTF-8. They are UTF-16. Usually. But in this case they are. I wanted to iterate over the set of resource strings and extract just the string I wanted. ...

Just about the time I believe the UTF-8 beast is in the cage, it escapes and runs amok. This AM, I started to deploy an update to the webapp on EC2. Seems that some of the static strings in the app contained UTF-8 encoded non-ascii characters. The java compiler barfed. “The heck?”, I thought. I just compiled the app on my MacBook. I checked the usual suspects (tomcat’s server.xml, JAVA_OPTS) but everything looked fine. However, when I looked at the code, it was indeed mangled. ...

I wanted to complement my bash directory crawler post with a bash array crawler example. Sometimes, it’s easier to jack a list of identifying tokens into an array and process them rather than to build an end-to-end script with database access. For this contrived example, I grab a list of UUID from MySQL with a simple SQL statement. mysql> SELECT id, uuid FROM icons; +-----+--------------------------------------+ | id | uuid | +-----+--------------------------------------+ | 1 | fe0b16ed-3369-4dda-8e60-faffb966375d | | 3 | 82bfcbc2-84a2-4ca7-914b-13172b94feb6 | | 6 | ab5e7265-3698-4205-b081-e6aec528fee2 | | 11 | 4b6ca26b-c6ed-494f-aeb4-9bf369e2d465 | | 19 | e7cc807b-7f15-46fa-b1c5-85d1f1050155 | +-----+--------------------------------------+ 5 rows in set (0.00 sec) Next, jack the tokens into an array and simply crawl over the tokens. ...

I’ve been building systems lately on Amazon’s Elastic Compute Cloud (EC2). At first, I was only interested in Amazon’s Simple Storage Solution (S3) after seeing the SmugMug slide show. I hadn’t really considered using EC2 since we had more servers in colocation than I really needed. But I had a file storage problem. When you have a thousand files, you stick them in a directory. When you have a million files, you cannot simply stick them in a single directory. You distribute them across multiple directories. What a PITA. ...

Currently, popular filesystems (ext3, hfs+) have a practical limit on the number of files and directories you can store in a single directory. Certainly, most of the unix command line tools will not work once you exceed some magic threshold. In my experience, 10,000 files and or directories is the practical limit. So what do you do when you have 1,000,000 XML files to process? I had this very problem recently. Fortunately, the problem was simplified as each file belong to one of 27,000 categories. ...