by Jim Bahn, Astute Networks

Leah Schoeb of The Evaluator Group, wrote a nice piece about storage benchmarking that we highly recommend. It appears in the January issue of Storage Magazine http://bit.ly/113klb2, and at http://bit.ly/YPgHQv.

I won't reiterate her article here, but I would like to simply point out that the specs many vendors throw around don't come with the appropriate caveats.  They don't mention block size, or special caching, or what network protocols were used. And if you're like me and you grew up in the HDD world, be prepared for a different set of rules for measuring and benchmarking solid state storage.

This specmanship reminds me of an old joke in the high fidelity music industry awhile back.  Vendors would claims lots of "watts" as a way to measure and compare their equipment, without explaining how it was measured, against what resistance for instance. When I asked my old boss how a particular manufacturer could claim such a large number of watts, he replied that they used the "ILS" standard.  By "ILS", he jokingly meant "If Lightning Strikes". I feel the same way when I read the incredible claims by some of my solid state marketing brethren.

More than one vendor claims million IOPS numbers by cobbling together science experiments that include all kinds of oddball multi-unit configurations, including the use of InfiniBand as the storage network. Does your shop run InfiniBand?

If you really want to compare performance among two or more solid state flash vendors, you need to run them side by side in your environment. This is not a difficult or disruptive task and Astute will send a ViSX storage appliance to any qualified end-user who wants to evaluate one for 30 days for free. That's how confident we are in our performance benchmarks.

Check out Leah's article and be prepared.

• A Reaction to the iPad
• Tax Cut Deal: 51 Million Will Take Home Less
• Uncompromising Actress Kim Cattrall Shows Women They Can Have it All With I Can't Believe It's Not Butter!® Spread in the Brand's Newest Campaign