the SSD Buyers Guide - click to see article
SSD buyers guide
the fastest SSDs - click to read article
the fastest SSDs
the problem with flash SSD  write IOPS
the problem with
write IOPS in FSSDs

storagesearch.com

by Zsolt Kerekes, editor

storage search
11 years - "leading the way to the new storage frontier"

Can you trust flash SSD specs & benchmarks?

published July 9, 2008 - we warned you first!

storage test equipment - click for vendor list
Storage Testers & Analyzers
on STORAGEsearch.com
Megabyte liked to keep his systems
in peak condition.
RamSan-20  very fast PCIe SSD from Texas Memory Systems
RamSan-20 very fast 450GB PCIe SSD
from Texas Memory Systems
the SSD Buyers Guide
flash SSD Jargon Explained
Are MLC SSDs Safe in Enterprise Apps?
the Problem with Write IOPS - in flash SSDs
Data Integrity Challenges in flash SSD Design
How Bad is - Choosing the Wrong SSD Supplier?
Z's Laws - Predicting Future Flash SSD Performance
Clarifying SSD Pricing - where does all the money go?
Why Consumers Can Expect More Flaky Flash SSDs!
Calling for an End to Unrealistic SSD vs HDD IOPS Comparisons
the pros and cons of using ASAPs - (Auto-tuning SSD Accelerated Pools of storage)
One of the things I've noticed is that the published specs of flash SSDs change a lot -from the time products are first announced, then when they're being sampled, and later again when they are in volume production. And flash SSD performance in real applications can be considerably lower than predicted from published benchmarks due to common"halo" errors which occur when using test suites designed for HDDs / RAM SSDs without understanding and changing the implicit assumptions which have been built into the legacy test set ups.

Sometimes the headline numbers get better, sometimes they get worse. There are many good reasons for this, because unlike memory devices, SSDs are complex systems in which software and controller hardware can all play a part in shaping the characteristics of a device. Tweaks in the controller algorithms can have any of the following major effects:-
high reliability flash SSDs  for embedded and high reliability servers
If you're a systems integrator / oem who has designed flash SSDs into a particular application - then it's unlikely that you chose your favored models at random. It's much more likely that you carefully evaluated products against a wish list of characteristics which include:-
  • performance,
  • environmental tolerance,
  • power consumption and
  • longevity.
These factors play as big a role as the obvious ones:-
  • capacity,
  • form factor,
  • host interface,
  • price and
  • security of supply.
But the flash SSD market is very volatile.

How can you be sure that the products which are going into your production systems are the same (or similar enough) to what you tested?

It's possible that after you did all your qualification testing that the original SSD oem did some "improvements" which they may not necessarily tell you about - because they make the product "better". But what if the new SSD controller chip does work faster - but puts more stress on your limited battery budget? Or what if the SSD oem has switched suppliers of memory or power management chips and the overall product fails to operate reliably over the full range of temperature you need?

Worse still - maybe you can't get the original product at all. To keep your production line going you have to stuff slots with products that your distributor suggests from companies you've never heard of before.

Many of these problems have been around in the electrinics industry in past decades. But in 2008, 2009 and maybe 2010, the unique characteristics of the flash SSD market mean that the risks are a lot worse.

Many SSD oems haven't been in the market very long. But because they make attractive products you can't afford to ignore them. Although some oems have been in the industrial or military markets for years - and do test their products and do inform you when those specs change - when their demand surges and products go on allocation - you still face the risks of switching suppliers to guarantee your own product's continuity.

And here's another thing to worry about.

Can you be sure - for example that the flash SSDs your buyer has bought at such a good price from an alternate source really are SLC? - and not MLC or SLC/hybrids. This is such a new market that you can't be sure that the supplier's SSD product manager (who may have been in the flash SSD business for less than 2 months) understands the intricate concerns you have - or what your questions mean.

One solution to protect yourself - may be to do much more sample testing of incoming product. Or if your volumes aren't high enough to justify the capital expense - another option might be to ask your distributor to do the testing for you.

The flash SSD market opens up tremendous opportunities for new products and systems which leverage that technology. But due to the diversity of products in the market and lack of industry standards - it's got tremendous risks as well.

Paying proper attention to compliance testing and quality assurance will make the difference between the market success or failure of many new SSD based products.

...Later:- Sometimes the "specs" don't tell you anything meaningful about the product's performance at all - particularly in the case of write IOPS tests for flash SSDs.

I've seen several published documents in which the measurements of such parameters appear to have been done incorrectly.

Because there isn't widespread market experience of flash SSDs - it's easy to fall into the trap of running tests which were originally designed for hard disks or RAM SSDs and not realise that some of the inbuilt assumptions or test parameters may be inappropriate.

It makes little or no difference to a small block size random write test on an HDD or RAM SSD whether the media being written to already contains data.

The performance impact of fragmentation on HDD performance is well known. Flash SSDs don't suffer from a dropoff in performance due to fragmentation - but there can be a similar performance droppoff over time due to the lower availability of pre-erased blocks. So when benchmarking a new flash SSD (which is initially erased) it's important to take this factor into account.

Some high performance lash SSDs have a background process which manages and tries to maximize the availability of pre-erased blocks. How well that works determines the sustained random write IOPS figure in 24x7 enterprise applications.

That's why when testing a flash SSD for 24x7 enterprise applications you must make sure that when the benchmark window begins - the disk is already full. Otherwise you get a "halo" effect (similar to that caused by cache hits in traditional server benchmark tests) in which the pool of pre-erased flash storage makes the SSD "appear" to operate much faster than it will be in a real application after it has been running for days, weeks or months.


About a year later:- STEC published this useful white paper - Benchmarking Enterprise SSDs (pdf). It reiterates many of the points originally raised in this StorageSearch.com article.
Targa Series 4 - 2.5 inch SCSI flash disk
Removable Military Solid State Disks
from Targa Systems
.
is the SSD Performance Deterministic?

Editor:- March 6, 2009 - if you know that enough readers are confident about SSD jargon - it paves the way to discuss weighty concepts.

Pliant Technology has neatly brought together the themes of SSD jargon and benchmark reliability in a blog which asks - is the performance deterministic? - among many other important questions.
.
Trident SSDs
advanced rugged solid state drives
from Trident Space & Defense
.
EDN Publishes Benchmarks on Intel's Fast Flash SSD

Editor:- November 17, 2008 - EDN has published benchmarks comparing Intel's new fast SLC flash SSDs with various other SATA flash SSDs and HDDs.

The author, EDN Senior Technical Editor Brian Dipert, has also published useful raw results on R/W IOPS and throughput.

However Dipert voices some concerns about the validity of these tests on flash SSDs.

"Since the data sets I wrote to the drives were fairly small in comparison to the SSDs' total available capacities, and since the drives were freshly formatted at the beginning of the tests, I suspect that the file patterns I wrote to the SSDs didn't trigger much if any media management. But sooner or later under normal usage conditions, the drive controller will need to begin tackling..."

The halo effect caused by invalid hard disk benchmark assumption were discussed in my article Can you trust your flash SSD's specs?
.
SiliconDrives from SiliconSystems
2.5" SiliconDrives
from Western Digital
.
Benchmark Reviews Confirms flash SSD Performance Uncertainty Principle
Editor:- January 4, 2009 - Benchmark Reviews has recently published an article - "SSD Benchmark Performance Testing."

This provides experimental confirmation of something I warned about in an earlier article - Can you trust flash SSD specs & benchmarks? - namely that hard disk based performance software provides unreliable results for SSDs.

The author, Executive Editor, Olin Coles (who has published many disk benchmarks) concludes "I warn readers to regard SSD reviews with a high degree of caution".

He says he's going to stop using certain test suites, and the discovery that he can't trust all the results which he has worked hard to collect and publish made him feel sick. Coles is a very experienced benchmark tester - and his article makes interesting reading. ...read the article
.
Another Publications Reports Unrepeatable Numbers from Repeated SSD Benchmarks

February 13, 2009 - an article published on PCPerspective.com reported declining performance in successive benchmarks of Intel's fast SSD.

The article mistakenly (in my view) attributes this to fragmentation. But the more likely cause is halo effects and poor garbage collection performance.
.
Data Integrity Challenges in flash SSD Design
Editor:- October 12, 2009 - StorageSearch.com published a new article - Data Integrity Challenges in flash SSD Design - written by Kent Smith Senior Director, Product Marketing, SandForce.

Reliability is the next new thing for SSD designers and users to start worrying about. The faster you make an SSD go - the more you have to engineer data integrity to eliminate the risk of "silent errors." ...read the article

Here are some more popular SSD related articles on storagesearch.com

SSD news
the Fastest SSDs
the SSD Buyers Guide
After SSDs... What Next?
Flash SSDs / RAM SSDs
What's a Solid State Disk?
the Top 10 SSD Companies
Increasing Flash SSD Reliability
Data Recovery from Flash SSDs?
How Solid is Hard Disk's Future?
Can you trust your flash SSD specs?
Is the SSD Market Recession-Proof?
30 Years of SSDs - SSD Market History
Why Seagate will Fail the SSD Challenge
the 10 biggest storage companies in 2012?
Are MLC SSDs Safe in Enterprise Apps?
SSD Myths and Legends - "write endurance"
RAM SSDs versus Flash SSDs - which is Best?
Flash Memory vs. Hard Disks - Which Will Win?
Using SSDs to Boost Legacy RAID Performance
3.5" Terabyte SSDs with Gigabyte / S Performance
Hybrid Storage Drives - winners, losers and maybes
Flash vs DRAM Price Projections - for SSD Buyers
War of the Disks: Hard Disk Drives vs. Flash SSDs
SSDs Pushing the Envelope in Blade Server Design
Z's Laws - Predicting Future Flash SSD Performance
Calling for an End to Unrealistic SSD vs HDD IOPS Comparisons

storage search banner

STORAGEsearch Solid state disks storage news Hard drives Disk to disk backup SSD Market History
STORAGEsearch is published by ACSL