I’ve been pensive of late and frankly the weather is just too damn nice for that. Still, there is something gained by wrapping up from time to time in a blanket of introspection. I’m sharing here my pensive playlist.
You’d think in this day and age that modern operating systems, especially OS X, would be set for UTF8 handling by default. Not so. My previous post, centos l10n problem, showed that CentOS defaults to set its locale LANG as POSIX rather than UTF8.
Mac takes the lunacy one step further. Or should I say one step backwards in time.
I use maven2 as my build manager. Normally, I ignore the stream of info at the beginning of a build, Either it succeeds (yeah) or it fails. Either way, I’ve been more interested in seeing the end result; You know, those last few lines rather than the first few lines.
One day, I started tracking down all the warnings and errors which popped up during maven builds and tomcat startups. I noticed this one.
$ mvn -Pdevelopment clean compile package war:inplace
[INFO] Scanning for projects...
<!-- snip -->
[WARNING] Using platform encoding (MacRoman actually) to copy↩
filtered resources, i.e. build is platform dependent!
<!-- snip -->
[INFO] ----------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ----------------------------------------------------------------
[INFO] Total time: 8 seconds
[INFO] Finished at: Tue Apr 07 23:40:06 PDT 2009
[INFO] Final Memory: 26M/63M
[INFO] ----------------------------------------------------------------
If you’ve ever had to trace down all the UTF8 failure points in a system then you know this maxim: “Suffer not a UTF8 Failure to Live.” Once you have a failure point, Latin1 (or worse in this case–MacRoman) will leak into your database and rot your data like a cancer.
I really should hunt down the BSD system configuration equivalents to Linux but here’s a solution that is quick and easy: add a project.build.sourceEncoding element and a project.reporting.outputEncoding to your pom.xml.
<project
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 ↩
http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>mywebapp</artifactId>
<packaging>war</packaging>
<version>1.21</version>
<name>mywebapp</name>
<properties>
<project.build.sourceEncoding>
UTF-8
</project.build.sourceEncoding>
<project.reporting.outputEncoding>
UTF-8
</project.reporting.outputEncoding>
</properties>
<!-- snip -->
</project>
Run maven again to verify the fix.
$ mvn -Pdevelopment clean compile package war:inplace
[INFO] Scanning for projects...
<!-- snip -->
[INFO] Using 'UTF-8' encoding to copy filtered resources.
<!-- snip -->
[INFO] ----------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ----------------------------------------------------------------
[INFO] Total time: 7 seconds
[INFO] Finished at: Tue Apr 07 23:48:17 PDT 2009
[INFO] Final Memory: 25M/60M
[INFO] ----------------------------------------------------------------
I really do want to understand the vagaries of OS X (relative to Linux) but I’m eternally short on time. I suspect that is our lot, all of us.
Whose woods these are I think I know.
His house is in the village though;
He will not see me stopping here
To watch his woods fill up with snow.
My little horse must think it queer
To stop without a farmhouse near
Between the woods and frozen lake
The darkest evening of the year.
He gives his harness bells a shake
To ask if there is some mistake.
The only other sound's the sweep
Of easy wind and downy flake.
The woods are lovely, dark and deep.
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep.
--Robert Frost
It had always bugged me that I got this warning on my OS X, MacBook Pro development system. (Well, it’s really an INFO, not a WARN but it bugged me nonetheless.)
Apr 8, 2009 4:10:55 PM org.apache.coyote.ajp.AjpAprProtocol init INFO: Initializing Coyote AJP/1.3 on ajp-8009 <!-- snip --> Apr 8, 2009 4:10:58 PM org.apache.tomcat.util.net.AprEndpoint allocatePoller INFO: Failed to create poller with specified size of 8192 <!-- snip --> Apr 8, 2009 4:10:58 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 2703 ms
I never saw this issue on my CentOS production servers so I figured it must be a BSD thing. It any rate, I recently tracked down a solution: add a pollerSize attribute to Tomcat’s connector element in server.xml.
<Connector pollerSize="1024" port="8009" URIEncoding="UTF-8" enableLookups="false" redirectPort="8443" maxPostSize="104857600" protocol="AJP/1.3" />
For a full listing of my server.xml, see UTF8 JDBC on Tomcat.
I suppose I could have mucked with OS X’s limits but somehow this solution seemed less problematic. I’m not serving to the web from my laptop so I feel comfortable lowering the pollerSize. Besides, I had two years of history in which a problem never arose even though tomcat failed to create the poller… I’ve had a couple of months now in which the poller has been created and works fine.
I’ve had opportunity to once again visit the UTF8 chain of failure and thought I’d write about it. If for no other reason, it’s easier for me to find my notes when I shove them into a blog entry.
I previously wrote about UTF8 on Tomcat. I pointed out that I needed to add an attribute to the connector element so that the mod_jk connection would be UTF8-ified. I neglected to also point out that I needed to UTF8-ified the database connection.
jdbc:mysql://localhost/mywebapp?useUnicode=true&characterEncoding=utf8
I’ve included my entire Tomcat server.xml file to illustrate just where this stuff goes. I don’t use port 8080; all my traffic comes through the mod_jk connector. Also, I don’t claim that I know what everything in my config file does. Most of everything: yes. All of everything: no.
I do claim that this server.xml has given me trouble free service for more than two years.
<Server port="8005" shutdown="SHUTDOWN">
<Listener
className="org.apache.catalina.core.AprLifecycleListener" />
<Listener
className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener
className="org.apache.catalina.mbeans↩
.GlobalResourcesLifecycleListener" />
<Listener
className="org.apache.catalina.storeconfig↩
.StoreConfigLifecycleListener"/>
<GlobalNamingResources>
<Environment
name="simpleValue"
type="java.lang.Integer"
value="30"/>
<Resource
name="UserDatabase"
auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector
pollerSize="1024"
port="8009"
URIEncoding="UTF-8"
enableLookups="false"
redirectPort="8443"
maxPostSize="104857600"
protocol="AJP/1.3" />
<Engine name="Catalina" defaultHost="www.mywebapp.com">
<Host
name="www.mywebapp.com"
appBase="webapps"
debug="4"
unpackWARs="true">
<Alias>www.mywebapp.com</Alias>
<Valve
className="org.apache.catalina.valves.AccessLogValve"
directory="logs/mywebapp"
prefix="access."
suffix=".log"
pattern="common"/>
<Logger
className="org.apache.catalina.logger.FileLogger"
directory="logs/mywebapp"
prefix="host."
suffix=".log"
verbosity="debug"
timestamp="true"/>
<Context
path=""
docBase="mywebapp"
debug="4"
reloadable="true">
<Logger
className="org.apache.catalina.logger.FileLogger"
directory="logs/mywebapp"
prefix="context."
suffix=".log"
timestamp="true"/>
<Resource
name="jdbc/mywebapp"
auth="Container"
type="javax.sql.DataSource"
driverClassName="com.mysql.jdbc.Driver"
username="somename"
password="somepass"
url="jdbc:mysql://localhost/mywebapp?useUnicode=true↩
&characterEncoding=utf8" />
<Resource
name="mail/Session"
auth="Container"
type="javax.mail.Session"
mail.smtp.host="localhost" />
</Context>
</Host>
</Engine>
</Service>
</Server>
I’ve been working quite a bit with the rest plugin for Struts2. The really nice thing about this plugin is the way it cleans up Struts URLs. Makes them more rails-like. I chuckled when depressed programmer suggested that struts2 is “WebWork on drugs.” I hate struts2. I really do.
Anyway, I have stripped down an AccountController to show just the POST service. In reality, the create() method is wired to a middle tier service that authenticates username, password pairs then updates session attributes with member id and other bits of persistent session data I need.
// imports omitted
@Results({
@Result(
name = "success",
type = ServletActionRedirectResult.class,
value = "account")
})
public class AccountController extends ActionSupport
{
private String username;
private String password;
// getters/setters omitted
public AccountController() { }
public HttpHeaders index() { return notImplemented(); }
public HttpHeaders show() { return notImplemented(); }
public HttpHeaders edit() { return notImplemented(); }
public HttpHeaders editNew() { return notImplemented(); }
public HttpHeaders update() { return notImplemented(); }
public HttpHeaders destroy() { return notImplemented(); }
public HttpHeaders create()
{
int status = (username.equals("alice")
&& password.equals("restaurant"))
? HttpServletResponse.SC_ACCEPTED
: HttpServletResponse.SC_UNAUTHORIZED;
return new DefaultHttpHeaders().withStatus(status);
}
private DefaultHttpHeaders notImplemented()
{
return new DefaultHttpHeaders()
.withStatus(HttpServletResponse.SC_NOT_IMPLEMENTED);
}
}
Note that I only return HTTP headers; the body content will always be empty.
I have found curl invaluable for documenting the API. This is a simple case but consider a much more complicated system with dozens of URLs and each URL implements many of the HTTP methods (including PUT and DELETE).
Third party developers are the bane of the support engineer. First, few people read documentation. They skim the material and furiously code. When their software fails, they file a bug that the API is broken. Usually, the API isn’t broken; the developer simply did not understand the API.
I subscribe to the agile manifesto value of “working software over comprehensive documentation.” In my work, I have found that a few curl examples clears up most of these issues. For example, to exercise the create() method in the AccountController, simply post a form.
curl \ --request POST \ --include \ --url "http://ws.example.com/account" \ --form "username=alice" \ --form "password=restaurant" \ --cookie-jar "cookies" \ --cookie "cookies"
I like to add the “–include” flag as it displays some extra header information. When I get a support call, I have the developer trot out the “documentation” curl examples and open a bash shell. This, of course, drives the Windows guys nuts–to which I reply, “buck up.” We work through the exercise of getting the http request working with the curl example. Then a miracle occurs. The developer now has a working example on their machine from which to re-examine their code.
A final note. The “–cookie-jar” and “–cookie” parameters will handle cookies between the web server and your curl commands. In otherwords, you can login to a website and these parameters will store your authenticated session id in a file. The file in this example is named “cookies” but it can be legal filename. You can then make subsequent calls to URLs, passing the cookies (and, therefore, the session id) back up to the server.
For example, to upload your avatar picture to your new social network, first login using the curl command above. This establishes an authenticated session. Then post your picture using the curl command below, making sure you pass the cookies back up.
curl \ --request POST \ --include \ --url "http://ws.example.com/avatar" \ --form "avatar=@somepix.jpg" \ --cookie-jar "cookies" \ --cookie "cookies"
Finally, if you need to add a description, publish the curl command as part of a bash script. For example,
#!/bin/bash # 1. you must login before you can upload the avatar # 2. the web server will reject any avatar exceeding 2MB # 3. do not forget the '@' symbol, a common mistake # 4. do not forget to include --cookie and --cookie-jar curl \ --request POST \ --include \ --url "http://ws.example.com/avatar" \ --form "avatar=@somepix.jpg" \ --cookie-jar "cookies" \ --cookie "cookies"
Good luck!
I’ve been putting off building the software RAID10 on marmaduke. Today, I put it off no longer.
The server marmaduke has six storage devices (2 IDE and 4 SATA)
$ ls -1 /dev/hd? /dev/hde /dev/hdf $ ls -1 /dev/sd? /dev/sda /dev/sdb /dev/sdc /dev/sdd
The CDROM is attached as /dev/hde and a 300GB HDD as /dev/hdf on which I’ve installed CentOS 5.2. The four SATA drives will be used to build a RAID 10. I’ve read through a number of postings on how to build a software RAID. The cleanest, shortest and clearest of them is on tgharold.com.
First, create a node for the array.
# mknod /dev/md0 b 9 0
I chose md0 since it was available.
The parameter ‘b’ directs mknod to create a block (buffered) special file.
The parameter ‘9′ is the Major version. (huh?) Seems the correct parameter can be found in /proc/devices. (see also, centos docs)
$ cat /proc/devices | grep -e "md$" 9 md
The parameter '0' corresponds to the last digit in the device /dev/md0. tgharold.com points out that the digit used in the device name should be the same as the last parameter. Why? Dunno. Something to look up some day.
Second, partition the disks, each and every one.
# fdisk /dev/sda # fdisk /dev/sdb # fdisk /dev/sdc # fdisk /dev/sdd
I don't know why I set the boot flag. The important point is to set the ID to 'fd' which is 'Linux raid autodetect'.
Device Boot Start End Blocks Id System /dev/sda1 * 1 60801 488384001 fd Linux raid autodetect /dev/sdb1 * 1 36481 293033601 fd Linux raid autodetect /dev/sdc1 * 1 36481 293033601 fd Linux raid autodetect /dev/sdd1 * 1 36481 293033601 fd Linux raid autodetect
One of my drives is larger than the others. Three drives (Seagate ST3300620AS) where previously used in a RAID5. It's impossible to find these drives any longer so I picked up the closest match (Seagate ST3500630AS). It has a larger capacity but otherwise the specs match. In a RAID10, the larger drive's extra space (~200GB) will go unused.
I formatted the drives but perhaps it was unnecessary. I did this as a check on each drive before I began building the array. Didn't seem to hurt anything.
# mkfs.ext3 /dev/sda1 # mkfs.ext3 /dev/sda2 # mkfs.ext3 /dev/sda3 # mkfs.ext3 /dev/sda4
Third, time to pull the trigger. Let mdadm do the heavy lifting.
# mdadm \ --create /dev/md0 \ -v \ --raid-devices=4 \ --chunk=32 \ --level=raid10 \ /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
Fourth, whether you format the drive individually or not you must format the RAID.
# mkfs.ext3 /dev/md0
Finally, mount the RAID.
# mount /dev/md0 /mnt/xen
Sweet.
$ df -h
Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-LogVol00
287G 36G 237G 14% /
/dev/hdf1 99M 19M 76M 20% /boot
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/md0 551G 198M 523G 1% /mnt/xen
Now, where is that Xen tutorial?