Speak a new language
so that the world
will be a new world.
—Rumi

2010 sucked.

Don’t get me wrong. There was a lot good happened in 2010. But as a vintage, it sucked. Big donkey balls.

At 2010′s wake, we make New Year’s resolutions in hopes that doing so will wash the taste out of our mouths. It’s what we do. Like paying taxes in April. Creatures of habit, that’s what we are. One of those habits is making resolutions at year’s end.

Sometimes a resolution is vague…as in, “I will lose weight.” But being the innately masochistic sufferers that we are, a resolution usually details the burden…as in, “I will wake at 5am and work out at Gold’s Gym for 2 hours every workday morning.”

In either case, we are liars and know it. Still, we make the resolution. And still yet, we can’t get that donkey ball taste out of our mouths. But for a short while, we live the fantasy that we are doing something about our pathetic lives.

2010 sucked and my mouth tastes of donkey balls.

Such is the nature of being human, its essence intrinsic in our art. Maybe that’s why Country Music is so popular. It’s raw, authentic and touches on a universal truth: Life is Suffering.

I’m not making resolutions this year. No empty promises to the gods in hopes they shine a little fortune my way. Nope. This year, I’m going tactical. Search and destroy: looking past; looking present; looking future.

What does that mean? It means, I will not make a list of habitual patterns of behavior and how I will change them. What I will do am doing is practicing awareness of my life: lived, living, and to be lived.

• Which relationships are toxic?
• Which relationships are neglected?
• Which loose ends need attention?
• Which loose ends can I write off?
• What dreams have been tabled?
  • For more than a year?
  • For more than 2 years?
  • For more than 10 years?
• Where do I compromise?

Stuff like that.

Aristotle noted that, “We are what we repeatedly do.”

It’s a chicken and egg dilemma but Ari gets it right. We are creatures of habit.

Look, life happens each moment and in each moment we handle life. Spouting resolutions at years end doesn’t change a thing. It’s the myriad small changes at each moment during the year that alters a course. All we really have is now, the present moment. But that’s all you need. In each moment, you can see your whole life.

To see a World in a Grain of Sand
And a Heaven in a Wild Flower,
Hold Infinity in the palm of your hand
And Eternity in an hour… –W. Blake

To bend what you do in any moment is to bend the universe. It’s in each single moment that habits are broken and entire new worlds are born.

The closest thing I have to a resolution this year is to practice standing in each moment, eight sides open.

The petabyte is required not just for the finished product. It’s needed to store all the sensor and camera data as well. Okay. I accept that Weta needed 1PB. How does one go about creating a petabyte storage facility? What are the tradeoffs? How much does it cost to build and then to maintain?

I need to get this out of my head and free up some brain cycles to continue with my Seeds of Discontent series. This article is a sidebar.

Disclaimer: I’m using a server build derived from the Seeds of Discontent serial (see also, Seeds of Discontent: System Benchmarking Hardware). It isn’t a traditional server room server but it serves as a baseline talking point. Besides, I can get budgetary pricing off newegg.com. It’s good enough for the purposes of this discussion. I’m sure the staff at Weta (or ILM or Dreamworks or Pixar) would have more insight into what *really* works.

I’ll choose the 4U chassis for this exercise since it better approximates the airflow of a desktop chassis. The 50U rack is taller than usual but it allows for 12 4-U servers plus a switch.

In the Seeds of Discontent serial, the HDD array uses the cheapest drives available; The final size is not important. In this sidebar discussion, final HDD capacity *is* important. This is not a single desktop box but a compute and storage cluster.

Each server functions as a compute unit as well as a storage unit.

A commodity multi-core CPU together with two commodity PCI Express GPU subsystems comprise the compute unit. Primary storage and swap for the compute unit is a two-disk RAID 0 SSD. The GPU cards have their own RAM while the CPU has two triple-channel banks of DDR3.

The server also hosts an HDD array which is not private storage but part of a larger storage cluster. For this exercise, I arbitrarily choose GlusterFS.

I’ve budgeted $2,500 USD per server (less the HDD array).

(circa mid-November 2010)
case+PSU       $ 200
mainboard        230
SSD              135
SSD              135
DDR3 24GB        550
CPU              850
GPU              200
CPU              200
--------------------
             $25,000 USD
              $2,500 USD

In addition, I’ve budgeted $1,000 per rack and $1,000 per switch. A fully constructed rack sans HDD is $32,000. YMMV.

rack         $ 1,000
switch         1,000
servers       30,000 (2,500 * 12 servers/rack)
--------------------
             $32,000 USD

Let’s add the HDD to build out a petabyte cluster. Since I’m using the Asus Rampage III (admitedly not a server mainboard), the two GPU fully consume the PCI Express lanes. There aren’t any lanes left for a RAID card. The two SSD drives occupy the two SATA III channels leaving the seven SATA II channels for the HDD array. Each server then adds seven HDD (and each rack adds 84 HDD) to the cluster.

As seen from this list (circa mid-November 2010), the lower capacity drives are not the cheapest drives per gigabyte.

SKU               $USD  cnt    GB   $/GB $K/PB  HDD/PB
---------------   ----  ---  ----  ----- -----  ----
WD5002ABYS-20PK   1750   20   500  0.175  175   2000
WD7502ABYS-20PK   2500   20   750  0.167  167   1333
0A39289-20PK      2700   20  1000  0.135  135   1000
WD7501AALS-20PK   1550   20   750  0.103  103   1333
WD6401AALS-20PK   1300   20   640  0.102  102   1563
WD5000AADS-20PK    900   20   500  0.090   90   2000
0F10381-20PK       900   20   500  0.090   90   2000
WD1001FALS-20PK   1800   20  1000  0.090   90   1000
WD5000AAKS-20PK    880   20   500  0.088   88   2000
WD6400AARS          55    1   640  0.086   86   1563
WD7500AADS-20PK   1250   20   750  0.083   83   1333
ST3500418AS         40    1   500  0.080   80   2000
WD7500AADS          55    1   750  0.073   73   1333
WD20EVDS-20PK     2800   20  2000  0.070   70    500
0F10383-20PK      1400   20  1000  0.070   70   1000
WD10EALS-20PK     1350   20  1000  0.068   68   1000
WD10EARS            65    1  1000  0.065   65   1000
WD10EARS-20PK     1200   20  1000  0.060   60   1000
WD15EARS-20PK     1700   20  1500  0.057   57    667
ST31000528AS        50    1  1000  0.050   50   1000
ST31500341AS        60    1  1500  0.040   40    667

Downselect the cheapest drive at each capacity point greater than or equal to 1TB.

SKU               $USD  cnt    GB   $/GB $K/PB  HDD/PB
---------------   ----  ---  ----  ----- -----  ----
WD20EVDS-20PK     2800   20  2000  0.070   70    500
ST31000528AS        50    1  1000  0.050   50   1000
ST31500341AS        60    1  1500  0.040   40    667

I want to minimize the number of drives while minimizing costs. The Seagate 1TB drive is both more expensive per GB and requires more drives per PB than the Seagate 1.5TB drive. It is immediately eliminated. The competition is between the Western Digital 2TB and the Seagate 1TB drive.

If cost were the only issue, then the Seagate drive would win. If drive count were the only issue then the Western Digital drive would win. To get closer to an answer, let’s build out the storage cluster.

Petabyte Cluster with Just a Bunch of Disks

        ======= count ======  ========= cost ==========
Drive   HDD   Servers  Racks    HDD     Rack     Total
-----  -----  -------  -----  -------  -------  -------
2.0TB    500       72      6   70,000  192,000  262,000
1.5TB    667       96      8   40,000  256,000  296,000

Even though the 2TB drives cost $30,000 more than the 1.5TB drives, the total cluster cost is $34,000 for the 1.5TB drive choice. Furthermore, there is no redundancy to protect against drive failure. How likely is a drive to fail? It’s not just likely to happen. It will happen. If a single drive will on average fail in five years, then a single drive in a pool of 500 drives will on average fail in 1/100 of a year (or roughly two drives a week).

Without debate, I posit that it is not possible to make nightly backups of a petabyte storage cluster. Firstly, we’d need a second petabyte. Secondly, that’s a lot of data to move and the cluster needs to continuously run compute jobs (rendering). The solution is redundancy either through local machine (e.g., RAID 6) or through GlusterFS replication (analogous to RAID 10 but at the cluster level).

Petabyte Cluster with RAID-6

        ======= count ======  ========= cost ==========
Drive   HDD   Servers  Racks    HDD     Rack     Total
-----  -----  -------  -----  -------  -------  -------
2.0TB    700      100    8.3   98,000  268,000  366,000
1.5TB  1,087      156     13   56,000  416,000  472,000

Petabyte Cluster with RAID-10

        ======= count ======  ========= cost ==========
Drive   HDD   Servers  Racks    HDD     Rack     Total
-----  -----  -------  -----  -------  -------  -------
2.0TB  1,000      144     12  140,000  384,000  524,000
1.5TB  1,334      292     16   80,000  512,000  592,000

The drive count starts to really add leading to more frequent drive failures. With a RAID 10 (or the equivalent GlusterFS replication scheme), the operations team can expect to replace five to six drives per week.

But the larger question is, how many servers are needed for the compute cluster? What if rendering needed no more than 60 compute nodes? If we fixed the compute nodes count to 60, we would need to add more drives per server. For the sake of discussion, assume we could load 24 drives per server but that doubles the cost per server before including drive costs (i.e., 2 * $2,500 = $5,000 per server sans HDD).

Furthermore, assume we’re using GlusterFS replication for the storage cluster redundancy. This presses the drive count up but lowers the complexity of building and maintaining local RAID systems each with 24 drives.

Petabyte 60 Server Cluster with RAID-10

        ======= count ======  ========= cost ==========
Drive   HDD   Servers  Racks    HDD     Rack     Total
-----  -----  -------  -----  -------  -------  -------
2.0TB  1,000       60      5  140,000  160,000  300,000
1.5TB  1,334       60      5   80,000  160,000  240,000

There is a $60,000 capital cost difference between the two clusters. Drive failure rates are a third lower for the 2TB drive cluster but the larger drive costs more per GB.

Drive Failure Rate
(5 Year time to fail)
Petabyte 60 Server Cluster with RAID-10

       == failure cost ==   HDD   failrate  === month ===
Drive  HDD  labor  subtot  units   (days)   fails   cost
-----  ---  -----  -----   -----  --------  -----  ------
2.0TB  140     70  $ 210   1,000     1.825   16.4  $3,518
1.5TB   60     70    130   1,334     1.368   21.9   2,851

These number presumed that both drives had the same MTBF. Digging a bit further we find that the WD20EVDS claims 1 million hours MTBF and the ST31500341AS claims 750,000 hours MTBF. That is, the operations staff can expect the Seagate drives to fail at a rate 1/3 greater than that of the Western Digital drives.

Sidenote: The two drives are in different classes. The Seagate drive spins at a faster rate (7,200 RPM) and claims performance. The slower Western Digital drive (5,400 RPM) claims consistency and lower power. However, the slower rate is fine for the storage cluster which acts as secondary storage.

I will not attempt to sort the apples to oranges comparisons between the two drive manufacturers. I shall take my previous calculations and adjust the 1.5TB drive cost up by a third.

Drive Failure Rate
(Adjusted Fail Rate)
Petabyte 60 Server Cluster with RAID-10

                        original      adjusted
       fail    HDD   === month ===  === month ===
Drive  cost   units  fails   cost   fails   cost
-----  -----  -----  -----  ------  -----  ------
2.0TB  $ 210  1,000   16.4  $3,518   16.4  $3,518
1.5TB    130  1,334   21.9   2,851   29.2  $3,801

A reversal of recurring costs. Does it matter? No. Not really. In my opinion, it’s more important to minimize the day-to-day operation hassles. The three hundred bucks a month (one way or the other) is noise. The $60K difference in initial capital costs is significant but not as significant as reliable operations.

Rumor has it that both Seagate and Western Digital will soon release a 3TB drive.

Update 2010-12-17: Xbit Labs reports on Hitachi’s new sixth-generation perpendicular magnetic recording (PMR) which “enable 3.5″ hard drives with 4TB or even 5TB capacities.”

My final ponderings on this fantasy cluster looks at the impact of future HDD capacities. For this, I simply speculate on the cost per gigabyte. If you have pricing information more closely tied to reality, please let me know.

Drive Failure Rate
(unadjusted 5year rates)
Petabyte 60 Server Cluster with RAID-10 (or equivalent)

       $/GB   == failure cost ==   HDD   failrate  === month ===
Drive         HDD  labor  subtot  units   (days)   fails   cost
-----  -----  ---  -----  -----   -----  --------  -----  ------
5.0TB  0.100  500     70    570     400     4.564    6.6  $3,762
4.0TB  0.090  360     70    430     500     3.651    8.2   3,526
3.0TB  0.080  240     70    310     667     2.737   11.0   3,410
2.0TB  0.070  140     70    210   1,000     1.825   16.4   3,518
1.5TB  0.040   60     70    130   1,334     1.368   21.9   2,851

This isn’t the end of the tradeoff line. At some point, 3.5 inch HDD will yield to the 2.5 inch form factor. The larger larger drives just won’t be available. That dynamic will change the equation. GPU cards will become increasingly more capable. CPU core counts will increase. RAM costs decline. Fewer servers will be needed. Fewer racks. Less power. I’m sure in my naïveté I’ve underestimated much here. However, I do believe one day the entire data center used to build Avatar will fit inside a 40 foot shipping container. And then, some time later–but not much later–that compute power will shrink to fit in a 20 foot container. And so on. The important point is that the capital costs (exluding facilities) for this fantasy cluster is under a million dollars. And it gets cheaper by the day.

These are my notes made while looking for an Aikido dojo in the San Francisco Bay’s mid-peninsula region. Specifically the communities in and around Palo Alto, CA. (image: wikimedia.org)

[Update 2011-12-20: I have chosen to train at Aikido Center and started attending Zazen and Aikido classes in October 2011.]

I won’t get into the ‘why’ I’m looking for an Aikido dojo in this posting. Most of what I would say can be gleaned from Stan Wrobel’s excellent book, Aikido for Self Discovery: Blueprint for an Enlightened Life. I’ve included a short book review.

I made this handy google map of the Aikido dojos listed below. There is a KML file for the Google Earth crowd. Longitude and latitude coordinates were converted from the street addresses using capelinks.com.

I have included the Sensei for each dojo. It’s partly out of respect and partly out of a sense that the Sensei defines the nature of a particular dojo.

I met Doran Sensei once, briefly, some ten years ago. In those few minutes, he made a lasting impression. You cannot honestly talk about Aikido on the mid-peninsula without including Doran Sensei and his contributions.

I never met Kushida Sensei (he lives in Michigan) but I attribute his influence at Enbukan dojo. I trained for two years under a former student of Kushida Sensei and have respect for the schools under his watch.

The Aikido Center (aka Aikido Center of Mountain View) has been in its current location for many years. The dojo has a strong zen element. That is zen as in ‘zen practice’ not to be confused with ‘zen decor’.

Aikido of Mountain View has roots to the beginnings of Aikido in the area through Nadeau Sensei. I may be mistaken but I believe the dojo was once located on Charleston near what is now REI.

The Institute of Transpersonal Psychology (ITP) has a dojo but I’m under the impression that it is open only to students, faculty and staff. Ironically, I’m a Masters Program student at ITP but since I’m in the Global Program, I don’t attend classes on campus. I live nearby so maybe I will be eligible. In any event, if/when I find out more information, I’ll update the listing. Frager Sensei is also Dr. Robert Frager, co-founder (along with Dr. James Fadiman) of ITP. Aikido is an integral part of the ITP Resident Program.

I don’t know anything more about the remaining dojo than what I’ve read on their websites. I recommend reading their material first hand.

On the matter of dues (or tuition), I included the figures I found on their websites. I feel the amount of the dues is not really important. Note that all the schools charge about the same amount (hundred bucks a month, give or take). This is a really expensive part of the country; it is very expensive to operate a dojo here. My guess is that the dues keep the doors open. Barely.

Only the adult class schedule is given. Most schools have a separate children’s schedule on their website.

The following list is in alphabetical order by dojo name. The information reflects what was listed on each dojo’s respective website cira mid-November, 2010. Things change with time. Check with the school for the latest information.

Aikido Center
Shiohira Sensei

2560 Wyandotte Avenue, #A
Mountain View, CA. 94043

Monday      18:30-20:00        basic
            20:00-20:30        open
Tuesday     17:45-18:30        zazen ?
            18:30-20:00        basic
            20:00-20:30        open
Wednesday   18:30-20:00        beginners
            20:00-20:30        open
Thursday    17:45-18:30        zazen ?
            18:30-20:00        basic
            20:00-20:30        open
Friday      18:30-20:00        beginners
            20:00-20:30        open
Saturday    10:00-11:00        basic
            12:20-13:00        basic

$100/month

Aikido of Mountain View
Nadeau Sensei

1924 Plymouth Street
Mountain View, CA 94043

Monday      18:30-20:00        fundamental
Tuesday     18:30-20:00        basic
Wednesday   18:30-20:00        basic
Thursday    18:30-20:00        basic
Saturday    09:30-11:00        weapons
            11:00-12:30        basic
Sunday      11:00-12:30        basic

$100/month

Aikido West
Doran Sensei

3164 Bay Road
Redwood City, CA 94063

Monday      06:30-07:00        basics
            18:00-19:00        basics
            19:00-20:15        general
Tuesday     18:00-19:00        basics
            19:00-20:15        general
Wednesday   06:30-07:00        basics
            18:00-19:00        basics
            19:00-20:15        general
Thursday    18:00-19:00        basics
            19:00-20:15        general
Friday      18:30-20:00        general
Saturday    08:45-09:45        basics
            10:00-11:00        general
Sunday      08:30-09:45        weapons
            10:00-11:00        general

$95/month

Aikido@Noon
Hale Sensei

501 Laurel Street,
Menlo Park, CA 94025

Monday      12:00-12:50
Wednesday   12:00-12:50

$333/3months resident
 449/3months non-resident
(Through City of Menlo Park)

Enbukan Dojo
Kushida Sensei

209 West Evelyn Avenue
Mountain View, CA 94041

Monday      19:00-19:50        2
            20:00-20:50        3
Tuesday     18:15-18:55        intro
            19:00-19:50        1
            20:00-20:50        2
Wednesday   19:00-19:50        3
            20:00-20:50        2
Thursday    18:15-18:55        intro
            19:00-19:50        2
            20:00-20:50        1
Friday      19:00-20:30        open
            20:00-20:50        3

$90/month

Institute of Transpersonal Psychology
Frager Sensei

1069 East Meadow Circle
Palo Alto, CA 94303

?           ?                  ?

Tuition unknown

Rocky Valley Aikido
Heery Sensei
Tuazon Sensei

1211 Arbor Road
Menlo Park, CA 94025

Tuesday     19:00-20:00        St. Raymond School
Wednesday   17:30-18:30        Mitchell Park
Thursday    19:00-20:00        St. Raymond School

$60/month

Sierra Aikido
Levenbach Sensei
Bennett Sensei

2134 Old Middlefield Way, Suite A
Mountain View, CA 94043

Monday      19:00-20:00        weapons
Tuesday     12:00-13:00
Wednesday   19:00-20:00
Thursday    12:00-13:00
Friday      19:00-20:00
Saturday    11:00-12:00        weapons

$100/month

Stanford Aikido
Doran Sensei

Arillaga Family Sports Center
Wrestling Room
641 Campus Drive
Stanford, CA 94305

Monday      12:00-13:00
Tuesday     12:00-13:00
Wednesday   12:00-13:00
Thursday    12:00-13:00
Friday      12:00-13:00

$ 50/quarter  Student
 105/quarter  Stanford Faculty/Staff
 225/quarter  Non-Stanford Affiliated
  15/quarter  Anyone paying dues at Aikido West

It is with heavy heart that I say farewell to Master Chief Petty Officer Rich Dowdy. He passed Thursday, September 2, 2010 at the U.S. Naval Hospital at Balboa.

I married into the family thirteen years ago. The first time I met Rich Dowdy, he gave me the hairy eyeball. Somehow I passed muster, even if I was an old Army soldier. In time, I was accepted into the family.

Rich was a helluva fishing buddy. We could fish all day and not say a word. He and I agreed on most of the important things in life except the Army-Navy game (which we simply didn’t mention).

Below is a picture of Rich and me on the USS Midway and one of the very few pictures in which he is smiling. I know the ladders were hell on his knees that day but he wouldn’t let anyone know it. He was like that. Down in the engine room, Rich’s eyes simply lit up. Being something of a gearhead myself, I could have stayed all day while the Master Chief gave me a personal tour. This was sacred ground, the heart and soul of a great ship.

Rich Dowdy was a sailor who held firmly the line, a family man who did more than most, and a friend who shall evermore be missed. I’ll see you again on the far shore, Rich.

Fair winds and following seas.

I recently finished Paul Bloom’s intro psych course and thought I’d continue my survey with Jeremy Wolfe’s MIT 9.00: Introduction to Psychology. I’m only two lectures in and already I sense Dr. Wolfe’s class has heavier technical leanings. That’s not a bad thing. To read broadly expands awareness.

I tried comparing the syllabi from Yale and MIT. There is some overlap but there is far more unique content than not. At least that’s my impression comparing the two sets of reading assignments. I’ll know more when I finish the balance of the 23 lectures.

So far, I’m pleased. Dr. Bloom has a speaking style that stands apart; he is a hard act to follow. Listening to another lecturer rehash the same syllabus would invite comparison on delivery rather than on content. Happily, this isn’t the case. Even when discussing common material, the lecturers maintain unique courses. As an example, both lecturers talk about Thorndike’s work with cats and puzzle boxes. Even though they talk about the same experiment, the context created by the speakers gives each of their respective lectures a different cast.

Truffles (1996 – 2010)
Loyal and Gentle Friend

“Look in my eyes, Lord, and my sins
will play out on them as on a screen.
Read them all.

Forgive what you can, and send
me on my path. I will walk on,
until you bid me rest.”
—Shepherd Book

I’ve stumbled upon a new means of relaxation at Open Yale Courses. It sounds strange that recordings of university lectures are relaxing but I find they are. Paul Bloom’s PSYC 110: Introduction to Psychology is far more interesting than television. I’m currently on lecture ten of twenty.

His has an enjoyable speaking style; the hour flies by. A sidebar at the end of Dr. Bloom’s lecture on Freud exemplifies his sense of humor.

“One other thing on Freud–just a story of the falsification of Freud. I was taking my younger child home from a play date on Sunday and he asked me out of the blue, “Why can’t you marry your mother or your father?” Now, that’s actually a difficult question to ask–to answer for a child, but I tried my best to give him an answer. And then I said–then I thought back on the Freud lecture and so I asked him, “If you could marry anybody you want, who would it be?” imagining he’d make explicit the Oedipal complex and name his mother. Instead, he paused for a moment and said, ‘I would marry a donkey and a big bag of peanuts.’ [laughter] Both his parents are psychologists and he hates these questions and at times he just screws around with us.” [source]

The lectures are available on the Open Yale Courses website as audio, video and text transcripts. There are reading assignments, which I skip. (I’m currently taking courses elsewhere with considerable reading requirements.) One could probably get most of the value found in the original course if they completed the assignments but that’s not really my objective. For me, the lectures are firstly informative and secondly entertaining.

I originally downloaded the audio files from iTunes U but found that I had to move the audio out of iTunes U and into music so I could create a playlists. The lectures show up out of order in iTunes U and I cannot for the life of me puzzle out how to rearrange the order except as a playlist.

It’s nice to find a gem among the dredge that we know as the internet.

I’ve become increasingly disillusioned with broadcast and cable news outlets. News outlets have sadly taken distinctly partisan roles in society. I find their content specious at best and makes for very poor commentary. (One notable except is Deutsche Welle.)

I recently stumbled upon spiked, which itself has a bias but its bias pleasantly lacks the shrill insanity commonly found nowadays. It’s not news but rather commentary on newsworthy events and trends. Even those articles with which I disagree will often have a point two I hadn’t considered. I find this to be the real value.

“spiked is an independent online phenomenon dedicated to raising the horizons of humanity by waging a culture war of words against misanthropy, priggishness, prejudice, luddism, illiberalism and irrationalism in all their ancient and modern forms. spiked is endorsed by free-thinkers such as John Stuart Mill and Karl Marx, and hated by the narrow-minded such as Torquemada and Stalin. Or it would be, if they were lucky enough to be around to read it.” —From the spiked about page, www.spiked-online.com