Red Leopard

I’m trying to move from Tomcat 5.5.28 to Tomcat 6.0.26 for my Struts2-based webapp. I previously tried–unsuccessfully–to make the move to Tomcat 6.0.20 but had to roll back. There were problems with the underlying connection to mail that I didn’t have time to track down.

I use a standard stack.

$ cat /etc/redhat-release
CentOS release 5.4 (Final)

$ /usr/sbin/httpd -v
Server version: Apache/2.2.3
Server built:   Nov 12 2009 18:43:41

$ java -version
java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode)

$ ls -l /usr/local | grep tomcat
… tomcat -> ./src/apache-tomcat-6.0.26

Be forewarned: I’m jotting down my notes and this seems to be the easiest place for me to find them later. If you find something unclear or missing, leave a comment and I will look into it.

I habitually edit and format log data to cut down on the noise and improve readability. This torques the purists. I’m not a purist. I won’t change the essentials but I will eliminate—what I consider to be—unimportant details. For example, I will replace unimportant date stamps

Mar 11, 2010 4:30:33 PM

with a marker that’s less distracting

$TIME

I also will wrap output so it will fit nicely in my blog. When I wrap, I will use the marker “↩”.

As an example, the $CATALINA_HOME/logs/catalina.out log entry produced upon startup:

$TIME org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.20.

$TIME org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true],↩
accept filters [false], random [true].

$TIME org.apache.coyote.ajp.AjpAprProtocol init
INFO: Initializing Coyote AJP/1.3 on ajp-8009

$TIME org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 544 ms

$TIME org.apache.catalina.core.StandardService start
INFO: Starting service Catalina

$TIME org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.26

$TIME org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor host-manager.xml

$TIME org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor manager.xml

$TIME org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ws

$TIME org.apache.coyote.ajp.AjpAprProtocol start
INFO: Starting Coyote AJP/1.3 on ajp-8009

$TIME org.apache.catalina.startup.Catalina start
INFO: Server startup in 2131 ms

So far, so good.

You can see from the startup entry that I am using the Tomcat Native Library. There is a nice section in Tomcat: The Definitive Guide that explains why not to use the native library but I use it anyway.

You’ll also notice that I don’t listen on port 8080; Tomcat connects only to Apache httpd via mod_jk.

#  grep -E "8[0-9]{3}" /usr/local/tomcat/conf/server.xml
<Server port="8005" shutdown="SHUTDOWN">
port="8009" 

# cat /etc/httpd/conf/workers.properties
workers.tomcat_home=/usr/local/tomcat
workers.java_home=/usr/java/default
ps=/

worker.list=router

worker.worker1.port=8009
worker.worker1.host=localhost
worker.worker1.type=ajp13
worker.worker1.lbfactor=1

worker.worker1.redirect=worker2

worker.worker2.port=8009
worker.worker2.host=localhost
worker.worker2.type=ajp13
worker.worker2.lbfactor=1

worker.worker2.activation=disabled

worker.router.type=lb
worker.router.balance_workers=worker1,worker2

Everything else is pretty standard.

When I go to shut down, Tomcat 6.0.26 gives me a few surprise errors.

$TIME org.apache.coyote.ajp.AjpAprProtocol pause
INFO: Pausing Coyote AJP/1.3 on ajp-8009

$TIME org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina

$TIME org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal↩
  with key of type [null]↩
  (value [com.opensymphony.xwork2.inject.ContainerImpl$10@5a9b8ff9])↩
  and a value of type [java.lang.Object[]]↩
  (value [[Ljava.lang.Object;@1b0952e8])↩
  but failed to remove it when the web application was stopped.↩
  To prevent a memory leak, the ThreadLocal has been forcibly removed.

$TIME org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal↩
  with key of type [null]↩
  (value [com.opensymphony.xwork2.inject.ContainerImpl$10@1220b36])↩
  and a value of type [java.lang.Object[]]↩
  (value [[Ljava.lang.Object;@620e06ce])↩
  but failed to remove it when the web application was stopped.↩
  To prevent a memory leak, the ThreadLocal has been forcibly removed.

$TIME org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal↩
  with key of type [null]
  (value [com.opensymphony.xwork2.inject.ContainerImpl$10@1b1402c4])↩
  and a value of type [java.lang.Object[]]↩
  (value [[Ljava.lang.Object;@3ec19fbf])↩
  but failed to remove it when the web application was stopped.↩
  To prevent a memory leak, the ThreadLocal has been forcibly removed.

$TIME org.apache.coyote.ajp.AjpAprProtocol destroy
INFO: Stopping Coyote AJP/1.3 on ajp-8009

The question I have is, “Is this new or is the new listener included in the Tomcat 6 server.xml file reporting a previously missed problem?”

It turns out that Tomcat 6 has new listeners that detect this kind of problem.

“The new Tomcat 6 adds a series of listeners to prevent against memory leaks and one of them, is against ThreadLocals.”

“The new memory leak detection code in Tomcat has found an issue with an application that needs to be fixed.”

This seems to be a case of an existing problem which is just now being reported.

I should mention that I started tomcat and immediately shut it down. My webapp did not process any http requests between startup and shutdown. That doesn’t preclude my webapp from being the culprit, though.

I would be happier if Tomcat and/or Struts2 (Spring?) in a future release could eliminate the SEVERE notice. There’s something unsettling about ignoring serious warnings.

I ate too much last night. My wife had ‘girls night out’ which left me unsupervised as I made my dinner.

I powered through an entire package of Johnsonville Brats. Cause and effect… I was wide awake at 2AM, bloated as a poisoned pup. Ufda!

And, yes. It was worth it.

“What the hell”, says me? Good time to catch up on some random and completely unnecessary web browsing.

As it turns out, I learned something from Leo Babauta over on zenhabits.net. I had for years considered myself a minimalist. Whether other people agreed or not is irrelevant. It was my self-view. Leo brings up a good point, “[minimalism is] basically an extension of simplicity — not only do you take things from complex to simple, but you try to get rid of anything that’s unnecessary. All but the essential.”

It’s a nice distinction. I have some things that are unnecessary but I won’t jettison them until they becomes a nuisance. Not many things. Just some things. It’s more accurate to say, “I’m a practitioner of simplicity.”

Simplicity and minimalism are very closely related but not exactly the same. So, why split hairs? Because definitions are important. It’s how we see the world. Crisp, distinct definitions lead to a clear view of the world. Fuzzy definitions lead to a fuzzy view. Non-existent definitions lead to blind spots, things out there in the world that you simply cannot see.

I never thought about the importance of definitions much until 1995 when I read The Non-Designer’s Design Book by Robin Williams. Robin describes in chapter one—The Joshua Tree Epiphany—her experience on Christmas when she received a tree identification book. In this book was a description of a Joshua tree, a strange desert dwelling plant. She was sure she had never seen one until later that day. In the cul-de-sac where her parents lived were four houses with Joshua Trees in the front yard. Robin hadn’t seen her neighbors’ Joshua trees for thirteen years. Only when she had a definition—some way to distinguish Joshua trees from all other trees—was Robin able to see them.

Definitions allow us to distinguish between things and the distinctions bring _those_ things into view.

Distinctions matter. They affect one’s world view. But not all distinctions matter to all people. For some people, the difference between simplicity and minimalism may be irrelevant. Perhaps they are neither a practitioner of simplicity nor a minimalist and the difference between the two _is_ splitting hairs—in their world, in their experience. But in my world, in my experience, the difference _is_ important.

Connecting the dots. It behooves us to invest at least a modicum of interest on the definitions our friends and colleagues find important. It is the mutual set of distinctions which people hold that allows them to communicate with any degree of precision. Without a mutual set, it’s unlikely that any message will be heard. The words may be there but listener can’t hear them. Much like Robin couldn’t see the Joshua tree.

Sometimes a problem persists long enough—is an irritant long enough—that I’ll burn an entire Sunday morning simply out of spite. Today’s irrational time-waste went to solving “Directory index forbidden by Options directive.”

[marmaduke ~] $ cat /var/log/httpd/error_log \
 | grep '\[error\]' \
 | head -1
[Sun Dec 06 09:25:05 2009] [error] [client 192.168.2.29]↩
Directory index forbidden by Options directive:↩
/var/www/documentation/public_html/

I have a development server that I use to offload work from my laptop.

[marmaduke ~] $ hostname
marmaduke

[marmaduke ~] $ cat /etc/redhat-release
CentOS release 5.4 (Final)

Among other things, marmaduke hosts scads of apache virtual hosts, including documentation. I’ve wanted to index the documentation directory for quite sometime but never could get apache configured to auto index.

Every configuration problem has already been solved by someone else. A quick search (apache+allow+directory+index) yielded surprisingly consistent instructions.

These instructions yielded consistent failure for me. I simply could not get auto indexing to work. Until now.

My apache installation is a basic yum install.

[marmaduke ~] $ yum list | grep httpd | grep installed
httpd.x86_64           2.2.3-31.el5.centos    installed
httpd-devel.x86_64     2.2.3-31.el5.centos    installed
httpd-manual.x86_64    2.2.3-31.el5.centos    installed

I’ve extracted (below) a partial listing of relevant parts from the httpd.conf file.

I include configuration files from two directories: module configurations in /etc/httpd/conf.d/ and all virtual host configurations in /etc/httd/conf.d/hosts/.

For the most part, I leave httpd.conf untouched. Note that apache runs under the user and group named ‘apache’.

# /etc/httpd/conf/httpd.conf (partial listing)
Listen 80

Include conf.d/*.conf
Include conf.d/hosts/*.conf

User apache
Group apache

ServerName marmaduke:80
UseCanonicalName Off

DocumentRoot "/var/www/html"

<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>

<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

DirectoryIndex index.html index.html.var

IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable

NameVirtualHost *:80

Permissions are important.

I place my documentation directories in /var/www/documentation/public_html and the entire tree from /var/www/documentation downwards has the same user and group name. All directories have 755 permission and all non-directories have 644 permission.

[marmaduke ~] $ ll /var/www
drwxr-xr-x 3 kelly apache 4096 Dec  6 09:23 documentation

[marmaduke ~] $ ll /var/www/documentation
drwxr-xr-x 7 kelly apache 4096 Dec  6 11:51 public_html

[marmaduke ~] $ ll /var/www/documentation/public_html
-rw-r--r-- 1 kelly apache 5174 Dec  6 09:33 favicon.ico
drwxr-xr-x 2 kelly apache 4096 Dec  6 09:21 icecast-2.3.2
drwxr-xr-x 3 kelly apache 4096 Dec  6 12:00 mysql-5.0
drwxr-xr-x 4 kelly apache 4096 Dec  6 11:50 mysql-5.1
drwxr-xr-x 3 kelly apache 4096 Dec  6 11:51 mysql-5.4
drwxr-xr-x 3 kelly apache 4096 Dec  6 11:51 mysql-5.5

[marmaduke ~] $ ll /var/www/documentation/public_html/mysql-5.0
drwxr-xr-x 3 kelly apache 4096 Dec  6 10:05 en

My virtual host enables indexing through the Directory directive. So far, this is consistent with most of the instructions found through web search. Yet, auto indexing still didn’t work for me.

# /etc/httpd/conf.d/hosts/documentation.conf
<VirtualHost *:80>
  DocumentRoot /var/www/documentation/public_html
  ServerName documentation.site
  Options Indexes FollowSymLinks

  <Directory "/var/www/documentation/public_html">
    Options Indexes FollowSymLinks
    Order allow,deny
    Allow from all
  </Directory>

  RewriteEngine on
  RewriteLogLevel 1
</VirtualHost>

It turns out that there is a nastly little surprise in one of the configuration files that comes standard with the centos yum install.

Inside LocationMatch, there the Options directive turns off Indexes. From the apache documentation, “Regardless of any access restrictions placed in <Directory> sections, the <Location> section will be evaluated last…”

# /etc/httpd/conf.d/welcome.conf
<LocationMatch "^/+$">

  # this is the culprit!
  Options -Indexes

  ErrorDocument 403 /error/noindex.html
</LocationMatch>

Since LocationMatch is evaluated last, the -Indexes parameter disables options set in any Directory directive. Either changing the parameter -Indexes to Indexes or deleting the welcome.conf file will allow auto indexing (assuming the other configurations are correct).

Yesterday, I stumbled upon a blog that made me laugh. The truth be told, I have a snarky side—my evil twin, if you will. I keep it in check. Mostly. Some of you who will understand. Some of you won’t. It’s a Gemini thing.

My twin is fun but good luck putting it back in its cage. And believe me, for every dollar of fun you get, you’ll pay ten dollars in social disaster. Best to keep the twin in check.

But yesterday, I stumbled upon Ted Dziuba’s blog and—holy smokes—my twin was rattling its cage! Ted’s rant on unicode is spot on. I’ll further say, unicode is super-important only to people for whom unicode is super-important. If you’re backend services only understand ASCII then unicode is anti-important.

For example, YouTube is happy with 无法停下 as are most of the online music services. But that cost money, believe it. If you can make do with ASCII, you’ll save money and a lot of headache.

Oh, how did I stumble upon Ted’s place? Saw his scalability article in the Hot Links list on highscalability.

My twin rip snorted.

Below are several resources I used in writing a speech delivered this morning for Early Risers Toastmasters entitled, “Feedback Loops in Personal Practices.” For those who were unable to attend, the talk focused on somatic learning and the importance of personal practices. (I am interested in personal practices as access to metaprogramming.) Feedback loops are important to mitigate the risk adopting destructive practices or of improperly performing the practice. I concluded with a tie-in reference to Theo’s new project of video recording the clubs speeches (if the speaker requests it).

When I promised this morning that I would publish the list, I didn’t think it would be such an ordeal. But when I stared at the bald list of book titles and author names, I knew it was lacking. The soup just needed a bit more seasoning.

Of course, any speech draws upon a lifetime of experiences. These are the books pulled from the bookcase and stacked upon my desk from which I double-checked material.

Adele Westbrook and Oscar Ratti
"Aikido and the Dynamic Sphere"
ISBN: 978-0804832847

GOOGLE: "Aikido and the Dynamic Sphere"
SOURCE: http://books.google.com/books?id=AWAMSZfc97EC
AMAZON: http://www.amazon.com/dp/0804832846

Adele Westbrook studied philosophy at Columbia University and has since made a career in advertising and publishing. She is currently an executive for a New York City publishing company.

Oscar Ratti received a degree in classical studies and law from the University of Naples, where we was an intercollegiate Greco-Roman wrestling champion, as well as a member of the championship judo team. Mr. Ratti is a commercial illustrator, and he serves as a design consultant for traditional and web-based publications.

Ms. Westbrook and Mr. Ratti have together taught aikido in New York, working with youth groups at centers affiliated with the YMCA.

KELLY: “I’ve read a number of Aikido books and found this to be the most enjoyable and the most pertinent to somatic learning. It wasn’t written for that purpose so you’ll have to work at ’seeing the broader picture’. If you simply want a contextual basis for reading parts of Strozzi-Heckler’s ‘The Leadership Dojo’, I recommend speed reading the material. However, I invite you to participate in Aikido classes over an extended period, at least a year. My rationale is consistent with the following excerpt from the Institute of Transpersonal Psychology:”

ITP: “It may seem paradoxical to include martial arts practice as an important aspect to being a therapist. When we think of the martial arts, words such as, ‘opponent’, ‘defeat’, and ‘against’ often come to mind. However, Aikido differs from disciplines such as karate, tai chi, and even yoga because it emphasizes the importance of blending with your partner. In Aikido, as in therapy, it is necessary to read body language and understand the intention of the person with whom you are working. These are some of the fundamental reasons that ITP requires the study of Aikido for our Residential students.” [complete article]

Richard Strozzi-Heckler
"The Leadership Dojo"
ISBN: 978-1583942017

GOOGLE: "Richard Strozzi-Heckler" "The Leadership Dojo"
SOURCE: http://books.google.com/books?id=87a_giMS88UC
AMAZON: http://www.amazon.com/dp/1583942017

Richard Strozzi Heckler, PhD is the founder and President of Strozzi Institute. A nationally known speaker and consultant on leadership and mastery, he has spent more than three decades researching, developing, and teaching the practical application of Somatics (the unity of language, action, and meaning) to business leaders and executive managers.

EXCERPT: “300 repetitions produce body memory, which is the ability to enact the correct movement, technique, or conversation by memory. It’s also been pointed out that 3,000 repititions creates embodiment, which is not having to think about doing the activity–it’s simply part of who we are.”

KELLY: “This was the book that pulled a lot of the other material together for me. It is not an academic study and I wouldn’t use it alone as an authoritative source. It does present a coherent description of somatic learning as practiced by the author in his training business. I suggest reading this material after having studied the other references mentioned in this list.”

Tracy Goss
"The Last Word on Power"
ISBN: 978-0385474924

GOOGLE: "Tracy Goss" "The Last Word on Power"
AMAZON: http://www.amazon.com/dp/038547492X

Tracy Goss is President of Goss-Reid Associates, a management consulting firm based in Austin, Texas. She specializes in working with CEOs and their senior management teams, worldwide, to invent and strategically plan an “impossible future” and to “re-invent” themselves and their executive cadre to successfully lead their organization into that future.

EXCERPT: “Language is the only leverage for changing the context of the world around you. This is because people apprehend and construct reality through the way they speak and listen. Or, as Martin Heidegger put it, “Language is the house of being.”

KELLY: “This is the best publicly available source of Cylon (i.e., what drives us is a mechanical process controlled by our structures of interpretation) doctrine. It may not be as accessible to readers who have not participated in a Cylon-esque education program. The material becomes clearer through experience. If you only READ the material, that’s okay. You will benefit from even just a conceptual understanding before reading the other reference material. I include ‘The Last Word on Power’ because it shares many aspects with somatic learning while remaining, in many ways, incompatible with somatic learning. Puzzling out exactly where is educational.”

Malcolm Gladwell  [wikipedia.org]
"Blink"
ISBN: 978-0316010665

GOOGLE: "Malcolm Gladwell" Blink
AMAZON: http://www.amazon.com/dp/0316010669

Malcolm Gladwell is a British-born Canadian journalist, author, and pop sociologist, based in New York City. He has been a staff writer for The New Yorker since 1996. He is best known as the author of the books The Tipping Point (2000), Blink (2005), and Outliers (2008).

KELLY: “I like Gladwell’s use of Paul Ekman’s work on facial expressions. While Gladwell isn’t an academic researcher, his treatment of facial expressions is both entertaining and easy to remember. I also like Gladwell’s treatment of John Gottman’s work on ‘thin slicing’. I integrated Ekman and Gottman’s work and juxtaposed that against Strozzi-Heckler’s material on somatic learning and what arose was in interesting postulate: ‘The mind makes its thoughts real for the body and the body/experience programs the mind.’”

James Robbins
"Build a Better Buddha"
ISBN: 978-0892540655

GOOGLE: "James Robbins" "Build a Better Buddha"
AMAZON: http://www.amazon.com/dp/0892540656

James Robbins holds two graduate degrees, a Master’s degree in English literature from the University of Texas at Austin, and a master’s degree in professional counseling from Amberton University in Dallas. His first book of non-fiction, Build A Better Buddha, was published in 2003 by Nicolas-Hays, Inc. In 2004, Tony Robbins, world-renowned peak performance coach, personally selected this book as motivational reading for his elite, international group of Platinum Partnership clients. Better Buddha examines a cross-section of East and West, integrating aspects of Western psychology with Eastern philosophy.

KELLY: “I don’t even know where to begin. Most of what’s being said today has been said millennia ago. A good primer.”

K. Anders Ericsson  [wikipedia.org]
"The Making of an Expert"
Harvard Business Review, July-August 2007

GOOGLE: Anders Ericsson "The Making of an Expert" filetype:pdf
SOURCE: coachingmanagement.nl

Dr. K. Anders Ericsson is Conradi Eminent Scholar and Professor of Psychology at Florida State University who is widely recognized as one of the world’s leading theoretical and experimental researchers on expertise.

EXCERPT: “By now it will be clear that it takes time to become an expert. Our research shows that even the most gifted performers need a minimum of ten years (or 10,000 hours) of intense training before they win international competitions. In some fields the apprenticeship is longer: It now takes most elite musicians 15 to 25 years of steady practice, on average, before they succeed at the international level.”

KELLY: “The Ericsson ‘10,000 hours rule’ is often cited and often used out of context. The paper is easily accessible (i.e., not pedantic) and I believe it essential to judging whether another author’s reference of Ericsson’s work is legitimate.”

Paul Ekman  [wikipedia.org]

Paul Ekman is a psychologist who has been a pioneer in the study of emotions and their relation to facial expressions. He is considered one of the 100 most eminent psychologists of the twentieth century.[1] The background of Ekman’s research analyzes the development of human traits and states over time. He retired in 2004 as professor of psychology in the Department of Psychiatry at the University of California, San Francisco (UCSF).

John Gottman  [wikipedia.org]

John Gottman, Ph.D. is known for his work on marital stability and relationship analysis through scientific direct observations published in peer-reviewed literature. Dr. Gottman found his methodology predicts with 90% accuracy which newlywed couples will remain married and which will divorce four to six years later. It is also 81% percent accurate in predicting which marriages will survive after seven to nine years. Dr. Gottman is a Professor Emeritus of psychology at the University of Washington, and with his wife Dr. Julie Gottman now heads a non-profit research institute.

Albert Mehrabian  [wikipedia.org]

Albert Mehrabian (born 1939, currently Professor Emeritus of Psychology, UCLA), has become known best by his publications on the relative importance of verbal and nonverbal messages. His findings on inconsistent messages of feelings and attitudes have been quoted throughout human communication seminars worldwide, and have also become known as the 7%-38%-55% Rule.

KELLY: “I recommend at least reading through the summary material on wikipedia as Mehrabian’s results are often misconstrued. As noted, ‘It is emphatically not the case that non-verbal elements in all senses convey the bulk of the message, though this is how his conclusions are frequently quoted.’ Nonetheless, Mehrabian’s work is adds to understanding somatic learning, in my opinion, in that voice and body are an integral element in projecting one’s message successfully or unsuccessfully; it’s not enough to focus strictly on language acts.”

Rolling Stone reports that Mazzy Star’s “Sandoval confirms her [sic] and her bandmate David Roback haven’t called it quits and they are still working on their anticipated fourth album. But she declines to give many specifics. ‘It’s true we’re still together,’ she says. ‘We’re almost finished [with the record]. But I have no idea what that means.’”

An album. Really? A NEW album.

Just when you knew you couldn’t get any luckier. I thought the group was done. This is huge. Entire swathes of a person’s life are painted in poetry. If you’re lucky, you’ll find your poet. Nearly a decade of my life, starting in the early nineties, bask in the smokey blue light of Hope Sandoval’s lyrics and David Roback’s music. Pure poetry.

BLUE LIGHT
So Tonight That I Might See (1993)

  There's a blue light in my best friend's room
  There's a blue light in his eyes
  There's a blue light, yeah
  I want to see it, shine

  There's a ship that sails by my window
  There's a ship that sails on by
  There's a world under it
  I think I see it, sailing away

  I think it's sailing
  Miles crashing me by
  Crashing me by
  Crashing me by

  There's a world outside my doorstep
  Flames over everyone's heart
  Don't you see them shining
  I want to hear them, beating for me

  I think I hear them
  Waves crashing me by
  Crashing me by
  Crashing me by