This system uses most of the Rampage III resources. The two Radeon cards will completely consume the PCI Express lanes (2 x16). The 24GB DDR3 will fill the six DRAM slots.

There are 7 SATA II and 2 SATA III ports.

The two SATA III ports connect the two SSD drives as a RAID Level 0. All non-user data is served from this drive (e.g., the operating system and benchmark applications).

One SATA II port connects the DVD writer.

Six SATA II ports connect the six HDD as a RAID Level 0. The HDD were selected for their cache (64MB), interface (SATA III, though connected to SATA II), spindle speed (7200 rpm, not as fast as 10,000 or 15,000 rpm but much, much cheaper) and price (cheap).

The two video cards were selected for being current and for being cheap (sort of).

Not shown in the picture are four 1920 x 1080 LCD monitors. These monitors pivot making them nice for a development (4320 x 1920).

Item 1                                         $  60
Antec Three Hundred Computer Case
Black Steel ATX Mid Tower

Item 2                                           180
CORSAIR AX850 850W Power Supply

Item 3                                           230
ASUS Rampage III LGA 1366 Intel X58
Micro ATX Intel Motherboard
N82E16813131658

Item 4                                           880
Intel Core i7-970 Gulftown 3.2GHz LGA 1366
130W Six-Core Desktop Processor
L2 Cache: 6 X 256KB
L3 Cache: 12MB
BX80613I7970

Item 5                                           225
Item 6                                           225
Patriot Gamer Series (3 x 4GB)
SDRAM DDR3 1333 (PC3 10666)
PGS312G1333ELK

Item 7                                            70
Item 8                                            70
Item 9                                            70
Item 10                                           70
Item 11                                           70
Item 12                                           70
Western Digital 3.5" Caviar
640GB 7200 RPM SATA 6.0Gb/s
Cache: 64MB
WD6402AAEX

Item 13                                          134
Item 14                                          134
Crucial RealSSD C300 2.5" SSD
64GB MLC SATA 6.0Gb/s
CTFDDAC064MAG-1G1

Item 15                                           10
SilverStone Bay Converter
mounting bracket 3.5" to 2 x 2.5"
SDP08

Item 16                                          185
Item 17                                          185
ASUS Radeon HD 6850
EAH6850 DirectCU/2DIS/1GD5
1GB 256-bit GDDR5 PCI Express 2.1 x16
EAH6850

Item 18                                           11
HIS CrossFire Bridge Interconnect Cable
HCFBC4065

Item 19                                          250
Item 20                                          250
Item 21                                          250
Item 22                                          250
Acer 1920 x 1080 LCD Monitor
B243H

Item 23                                           25
LITE-ON 24X DVD Writer Black
iHAS424-98

I don’t know how long it takes to assemble the movie from source assets into a shipping product but I’ll venture it is quite a compute intensive process. I got the idea for this benchmark while reading on the sintel.org home page that the project is re-rendering the film for 4K and that it should be ready later this year (still a few months away). I didn’t know what 4K was so I looked it up. That lead me to compile a table of film formats and supported Frames per Second (FPS). I also appended three rows for HDTV of which only 720p and 1080p are real. 4320p is something bandied about as future but I couldn’t find material online that would lead me to believe it’s anything but a concept. I included it to define the upper envelope edge. (Note: different formats support different levels of FPS.)

Fomat   Width    Height    Frames per Second (FPS)
------- ------   ------  ----------------------------
2k 16:9 (2,048 × 1,152)  24, 25, 30, 50, 60, 100, 120
2K 2:1  (2,048 × 1,024)  24, 25, 30, 50, 60, 100, 120
2K ANA  (1,408 × 1,152)  24, 25, 30, 50, 60, 100, 120

3k 16:9 (3,072 × 1,728)  24, 25, 30, 50, 60
3K 2:1  (3,072 × 1,536)  24, 25, 30, 50, 60
3K ANA  (2,112 × 1,728)  24, 25, 30, 50, 60

4k 16:9 (4,096 × 2,304)  24, 25, 30
4K 2:1  (4,096 × 2,048)  24, 25, 30
4K HD   (3,840 × 2,160)  24, 25, 30
4K ANA  (2,816 × 2,304)  24, 25, 30

4.5k WS (4,480 × 1,920)  24, 25, 30

720p    (1,280 ×   720)  24, 25, 30, 50, 60
1080p   (1,920 × 1,080)  24, 25, 30, 50, 60
4320p   (7,680 × 4,320)  24, 25, 30, 50, 60

From this, I build a spreadsheet table of “Gigabytes per Second” of uncompressed data and converted the table into a set of data files compatible with gnuplot. Below are plots for 24 and 60 FPS for three formats.

I chose 24 FPS since it is a standard film frame rate. I excluded 25 and 30 FPS since they were so close to 24 FPS that they didn’t add any additional insight into designing the test. (25 and 30 FPS happen to be half of 50 and 60 FPS, which I discuss next). One datapoint to consider is that Blu-Ray supports 1080p at no greater than 24 FPS.

Just as the world is split between driving on the left or right hand side of the road, the world’s electrical power is split between 50 and 60 Hz. Neither is right or wrong but they are different and the two don’t mix well (like people driving on both the left and right side of the same road don’t mix well). I currently live in a 60Hz world so I biasedly chose the familiar but I could have easily chosen 50Hz. They are so close to each other in context of this exercise that the differences are negligible.

I chose two HD formats (4320p and 1080p) plus one film format (4K 16:9). The film format file size falls between the two HD formats. Note that 4K doesn’t support a frame rate obove 30 FPS and isn’t included in the 60 FPS graph.

Note that I haven’t included compression in this test as I believe it to be a different strain on the system.

My thoughts for the blender test was to establish a benchmark that would be inherently obvious and understandable to anyone, especially to those without any previous knowledge of video rendering or video formats. It’s more intuitive to say, “This system renders the video 1000 times slower than real time.” It also makes for a nice roadmap chart. Consider time on a linear x-axis and rendering ratio on a logrithmic y-axis.

A second roadmap milestone map could be what format/fps combination(s) achieve real time rendering status for a specific system on a specific date.

It would be interesting to see the performance of an ad hoc office cluster vs. a 40 foot shipping container of specialized hardware. If this sounds crazy, juxtapose any of the recent systems from Tom’s Hardware System Builder Marathon against the original IBM PC. (Or the ENIAC ;^)

In the conclusions section, I’d want to see analysis/conjecture about the test results regarding system bottlenecks. In an attempt to compare the systems, I’d also want to see the individual system price divided by the performance ratio. For example,

System A
Cost         $1200 USD
Ratio        0.0005245  (1830:1)
Cost/Ratio   2,196,000

System B
Cost         $1153 USD
Ratio        0.0005736  (1735:1)
Cost/Ratio   2,021,209

Cost Comparison
System A  : System B
    1,200 : 1,153
    1.040 : 1  (A is 4% more expensive than B)

Ratio Comparison
System A  : System B
    1,830 : 1,735
    1.055 : 1  (A is 5.5% slower than B)

Cost/Ratio Comparison
System A  : System B
2,196,000 : 2,021,209
    1.086 : 1  (A is 8.6% more expensive to
                to finish the same job as B)

If one starts to compare operating environments (Windows 7, OS X, Ubuntu, Fedora) on the same system and then repeats those tests across various systems (Dell, Apple, Whitebox), we start to get an apples to apples comparison in price, performance and price/performance ratios.

Note: the spreadsheet (numbers, exported to excel), data files, gnuplot scripts and images are available for download.

[update 2010-10-05: I just read at thenextweb.com that rendering Avatar requires 1 petabyte of drive space. That's in 3D but it still makes me wonder why there's a wide gap between my calculations of a hypothetical 2 hour movie and that of Avatar. The article at information-management.com gives a reference of 17.28 gigabyte/minute. At 166 minutes, this is 2.8 terabytes (a long way off from a petabyte). Perhaps there are a lot of intermediate stages for each frame which must be saved? Nevertheless, I believe the test is still interesting.]

I read a CPU benchmark this morning which got me wondering if there were a possiblity to benchmark on any operating system but Windows. I concluded for the kind of benchmarks seen on Tom’s–not really.

Many (most?) of the benchmarks rely on software only available on Windows. The platforms are varied while holding the software and operating system constant. This makes sense as Tom’s Hardware is, well, primarily a hardware site.

But what could we could we learn by holding the hardware constant and vary the operating systems? A lot, I believe.

One point I’ve noticed in reading Tom’s benchmarks over the years is the increasingly predictable nature of the results. Motherboards often perform within a few percentage points of each other. THere is some variance in different CPUs but rarely unpredicable. The conclusions often center around cost-performance comparisons to see if the premium parts are justified.

What really motivated me to write this blog entry was a video card benchmark. The two major vendors (AMD née ATI, NVidia) have been long locked in fierce battle. The benchmark seems to keep the software constant and only varies the video card hardware but the lines are not so cleanly drawn. Each video card has a unique architectural philosphy and design. Software modules are written to detect and take special advantage of these differences in video cards.

I argue that a systems approach might make for an interesting benchmark and would likely yield surprising results. Furthermore, such a benchmark would spark the desktop religious wars that have subsided in recent years (e.g., 68000 vs x86, PowerPC vs x86, IPX vs TCP/IP, RLL vs ESDI, Word vs. WordPerfect, Microsoft vs. Novell …). It seems the good fight has now moved on to smartphones and tablets. The desktop market has become docile.

My aim is to whack the hive. This is my vision.

1. This benchmark has a name: “Seeds of Discontent”

Why this name? It fits. How?

Ah. A little about me. My first job out of college was writing assembly language graphics routines used in medical imaging. On a 12MHz 80286. This introduced me to graphics hardware leading to a job at National Semiconductor (yes, National once developed graphics chips) and a deeper understanding of PC architecture. In the late nineties, I moved on to a small consumer electronics startup and eventually into pure software. I’ve seen the industry from several perspectives.

The early PC days placed importance on CPU speed for better performance and hardware integration (combining many chips into a single chip) for lower cost. The early days are gone and the funeral was the death of Comdex which happened a few years before the last Comdex. Those last few Comdex shows were zombies and those of you who were there know what I mean.

Nowdays, the big advances in hardware don’t spur the kind of religious fervor of old. It evokes a semi-interested “Oh, that’s kinda cool” response. (e.g., PATA to SATA, HDD TO SDD, dual core to quad core to six core). Interesting but not controversial.

This isn’t to say the hardware designers are sleeping. They are not. It’s just that hardware advancments alone won’t produce the kinds of dramatic performance improvements year-over-year that we saw during the eighties and nineties. The kinds of hardware advances we now see require specialized software to take advantage of those advances. And the industry knows it.

The greatest potential for seeing surprisingly results in a benchmark are not in swapping out CPUs or mainboards. The greatest potential lies in swapping out the operating systems. If there is one scrap of religiosity left in the industry, it’s in operating systems. As it happens, the OS is also the component that now has the biggest potential to influence the industry.

Which brings me to my point. I believe the right set of benchmark tests could pit OS vs OS and spur the kind of religious spat that leads to real progress. But with peace breaking out all over the industry, whipping up a bit of OS hooiganism might just whip up a real revolution. As David W. once told me, “Man’s best friend is his dogma.” Today, there is nothing more religious in the industry than operating systems.

2. “Seeds of Discontent” has a mission: I see you, OS

Chicken and egg.

Mulitcore CPUs and graphics engines both require specialized software to take advantage of the hardware. But today’s changing hardware is a moving target. It’s hard for a small software firms to write specialized hardware for an ever changing platform. But without software running on a platform (e.g., graphics), there is no “tie down” for a hardware vendor to stick with a stable API (I’m looking at you Nvidia, AMD).

In the multicore CPU arena, it takes a partner in the OS vendor to take full advantage of the multicore architecture. This isn’t to say mutithreaded software is held back by today’s OS but rather todays OS don’t do enough to create a furtile ground for multithreaded/multicore software development. (Yeah, the penguinistas are going ape-shit about now but I stick with my proposition.)

The real problem, as I see it, is that it takes a coordinated effort to make the necessary software development changes. No body wants to spend their money first. Nobody. Especially those with no money (the solo developer with no resources other than time.)

Disclaimer: I am a macinista.

Apple’s Snow Leopard (I have no information on Lion) arrived with much fanfare about Grand Central Dispatch, LLVM, OpenCL, blah blah blah and other multicore magic. I love the candy-coated icons for Core-X. But how has that translated into better software support for modern hardware? You don’t hear much about it (even if it has or hasn’t happened).

“Seeds of Discontent” is a set of benchmarks to provide an adequacy gap for each OS to buck up and meet the challenge of each other. Really. Can anyone name a benchmark today by which one OS can embarrass another? No.

The real value add an OS can deliver is hardware abstraction. (Unless you are the dominate OS -Microsoft–software developers are undermotivated to deliver software cutomized for specialize hardware under the OS.) I want OS vendors motivated to deliver better performance with cost-effective effort for software vendors. Not exlusively but in a multi-vendor environment.

Benchmarks give a measurable performance comparison between operating systems. And the operative word here is “System”. Given a fixed hardware platform, how do the various system’s perform. It’s not which comes first “chicken or egg,” what comes first is the benchmark and the “gap of shame.” Nobody wants to be last. Nobody.

3. Who?

To compare various Operating Systems presupposes a fixed hardware platform, (I say.) It also presupposes running the same software on all systems. Technically, that is impossible. Well, impossible except in a systems perspective.

Software makes OS calls to handle many functions. Mostly this is to abstract the hardware details from the OS. What’s needed then is software that has been compiled for each of the target systems. Since I’m interested in system performance (i.e., what you see when you use the system), I don’t care about how the software is written. If a package is not opitimized to use the special benefits offered by the OS, I don’t care. In my world, the experience is diminished.

So the one thing that is constant in a benchmark is the hardware. This isn’t such a problem for Linux and Windows but does present a problem for OS X. I want OS X included. Here lies a fork in the road. Does the benchmark use an Apple computer and install all OS on that machine? Or does the benchmark use a white-box PC and install all OS that machine?